issue_comments
10,495 rows sorted by updated_at descending
This data as json, CSV (advanced)
issue >30
- Show column metadata plus links for foreign keys on arbitrary query results 50
- Redesign default .json format 48
- Rethink how .ext formats (v.s. ?_format=) works before 1.0 48
- JavaScript plugin hooks mechanism similar to pluggy 47
- Updated Dockerfile with SpatiaLite version 5.0 45
- Complete refactor of TableView and table.html template 45
- Port Datasette to ASGI 42
- Authentication (and permissions) as a core concept 40
- Deploy a live instance of demos/apache-proxy 34
- await datasette.client.get(path) mechanism for executing internal requests 33
- Maintain an in-memory SQLite table of connected databases and their tables 32
- Ability to sort (and paginate) by column 31
- link_or_copy_directory() error - Invalid cross-device link 28
- Export to CSV 27
- base_url configuration setting 27
- Documentation with recommendations on running Datasette in production without using Docker 27
- Optimize all those calls to index_list and foreign_key_list 27
- Support cross-database joins 26
- Ability for a canned query to write to the database 26
- table.transform() method for advanced alter table 26
- New pattern for views that return either JSON or HTML, available for plugins 26
- Proof of concept for Datasette on AWS Lambda with EFS 25
- WIP: Add Gmail takeout mbox import 25
- Redesign register_output_renderer callback 24
- "datasette insert" command and plugin hook 23
- Make it easier to insert geometries, with documentation and maybe code 23
- Datasette Plugins 22
- .json and .csv exports fail to apply base_url 22
- Idea: import CSV to memory, run SQL, export in a single command 22
- Plugin hook for dynamic metadata 22
- …
id | html_url | issue_url | node_id | user | created_at | updated_at ▲ | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
1076662556 | https://github.com/simonw/sqlite-utils/pull/419#issuecomment-1076662556 | https://api.github.com/repos/simonw/sqlite-utils/issues/419 | IC_kwDOCGYnMM5ALJEc | codecov[bot] 22429695 | 2022-03-23T18:12:47Z | 2022-03-23T18:12:47Z | NONE | Codecov Report
```diff @@ Coverage Diff @@ main #419 +/-=======================================
Coverage 96.55% 96.55% Continue to review full report at Codecov.
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Ignore common generated files 1178484369 | |
1076652046 | https://github.com/simonw/datasette/issues/1670#issuecomment-1076652046 | https://api.github.com/repos/simonw/datasette/issues/1670 | IC_kwDOBm6k_c5ALGgO | simonw 9599 | 2022-03-23T18:02:30Z | 2022-03-23T18:02:30Z | OWNER | Two new things to add to the release notes from https://github.com/simonw/datasette/compare/0.61a0...main - https://github.com/simonw/datasette/issues/1678 - https://github.com/simonw/datasette/issues/1675 (now also a documented API) |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Ship Datasette 0.61 1174423568 | |
1076647495 | https://github.com/simonw/datasette/issues/1670#issuecomment-1076647495 | https://api.github.com/repos/simonw/datasette/issues/1670 | IC_kwDOBm6k_c5ALFZH | simonw 9599 | 2022-03-23T17:58:16Z | 2022-03-23T17:58:16Z | OWNER | I think the release notes are fine, but they need an opening paragraph highlighting the changes that are most likely to break backwards compatibility. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Ship Datasette 0.61 1174423568 | |
1076645636 | https://github.com/simonw/datasette/pull/1574#issuecomment-1076645636 | https://api.github.com/repos/simonw/datasette/issues/1574 | IC_kwDOBm6k_c5ALE8E | simonw 9599 | 2022-03-23T17:56:35Z | 2022-03-23T17:56:35Z | OWNER | I'd actually like to switch to slim as the default - I think Datasette should ship the smallest possible container that can still support extra packages being installed using |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
introduce new option for datasette package to use a slim base image 1084193403 | |
1076644362 | https://github.com/simonw/datasette/pull/1665#issuecomment-1076644362 | https://api.github.com/repos/simonw/datasette/issues/1665 | IC_kwDOBm6k_c5ALEoK | simonw 9599 | 2022-03-23T17:55:39Z | 2022-03-23T17:55:39Z | OWNER | Thanks for the PR - I spotted an error about this and went through and fixed this in all of my repos the other day: https://github.com/search?o=desc&q=user%3Asimonw+google-github-actions%2Fsetup-gcloud%40v0&s=committer-date&type=Commits |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Pin setup-gcloud to v0 instead of master 1173828092 | |
1076638278 | https://github.com/simonw/datasette/issues/1670#issuecomment-1076638278 | https://api.github.com/repos/simonw/datasette/issues/1670 | IC_kwDOBm6k_c5ALDJG | simonw 9599 | 2022-03-23T17:50:55Z | 2022-03-23T17:50:55Z | OWNER | Release notes are mostly written for the alpha, just need to clean them up a bit https://github.com/simonw/datasette/blob/c4c9dbd0386e46d2bf199f0ed34e4895c98cb78c/docs/changelog.rst#061a0-2022-03-19 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Ship Datasette 0.61 1174423568 | |
1075438684 | https://github.com/simonw/datasette/issues/1681#issuecomment-1075438684 | https://api.github.com/repos/simonw/datasette/issues/1681 | IC_kwDOBm6k_c5AGeRc | simonw 9599 | 2022-03-22T17:45:50Z | 2022-03-22T17:49:09Z | OWNER | I would expect this to break against SQL views that include calculated columns though - something like this:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Potential bug in numeric handling where_clause for filters 1177101697 | |
1075437598 | https://github.com/simonw/datasette/issues/1681#issuecomment-1075437598 | https://api.github.com/repos/simonw/datasette/issues/1681 | IC_kwDOBm6k_c5AGeAe | simonw 9599 | 2022-03-22T17:44:42Z | 2022-03-22T17:45:04Z | OWNER | My hunch is that this mechanism doesn't actually do anything useful at all, because of the type conversion that automatically happens for data from tables based on the column type affinities, see: - #1671 So either remove the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Potential bug in numeric handling where_clause for filters 1177101697 | |
1075432283 | https://github.com/simonw/datasette/issues/1671#issuecomment-1075432283 | https://api.github.com/repos/simonw/datasette/issues/1671 | IC_kwDOBm6k_c5AGctb | simonw 9599 | 2022-03-22T17:39:04Z | 2022-03-22T17:43:12Z | OWNER | Note that Datasette does already have special logic to convert parameters to integers for numeric comparisons like Though... it looks like there's a bug in that? It doesn't account for |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply 1174655187 | |
1075435185 | https://github.com/simonw/datasette/issues/1671#issuecomment-1075435185 | https://api.github.com/repos/simonw/datasette/issues/1671 | IC_kwDOBm6k_c5AGdax | simonw 9599 | 2022-03-22T17:42:09Z | 2022-03-22T17:42:09Z | OWNER | Also made me realize that this query:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply 1174655187 | |
1075428030 | https://github.com/simonw/datasette/issues/1671#issuecomment-1075428030 | https://api.github.com/repos/simonw/datasette/issues/1671 | IC_kwDOBm6k_c5AGbq- | simonw 9599 | 2022-03-22T17:34:30Z | 2022-03-22T17:34:30Z | OWNER | No, I think I need to use Even the details from the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply 1174655187 | |
1075425513 | https://github.com/simonw/datasette/issues/1671#issuecomment-1075425513 | https://api.github.com/repos/simonw/datasette/issues/1671 | IC_kwDOBm6k_c5AGbDp | simonw 9599 | 2022-03-22T17:31:53Z | 2022-03-22T17:31:53Z | OWNER | The alternative to using This feels a bit neater to me, but I still then need to solve the problem of how to identify the "type" of a column that I want to use in a query. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply 1174655187 | |
1074479932 | https://github.com/simonw/datasette/issues/339#issuecomment-1074479932 | https://api.github.com/repos/simonw/datasette/issues/339 | IC_kwDOBm6k_c5AC0M8 | simonw 9599 | 2022-03-21T22:22:34Z | 2022-03-21T22:22:34Z | OWNER | Closing this as obsolete since Datasette no longer uses Sanic. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Expose SANIC_RESPONSE_TIMEOUT config option in a sensible way 340396247 | |
1074479768 | https://github.com/simonw/datasette/issues/276#issuecomment-1074479768 | https://api.github.com/repos/simonw/datasette/issues/276 | IC_kwDOBm6k_c5AC0KY | simonw 9599 | 2022-03-21T22:22:20Z | 2022-03-21T22:22:20Z | OWNER | I'm closing this issue because this is now solved by a number of neat plugins:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Handle spatialite geometry columns better 324835838 | |
1074478299 | https://github.com/simonw/datasette/issues/1671#issuecomment-1074478299 | https://api.github.com/repos/simonw/datasette/issues/1671 | IC_kwDOBm6k_c5ACzzb | simonw 9599 | 2022-03-21T22:20:26Z | 2022-03-21T22:20:26Z | OWNER | Thinking about options for fixing this... The following query works fine:
If someone clicks on "View and edit SQL" from a filtered table page I don't want them to have to wonder why that But... for querying views, the So one fix would be to get the SQL generating logic to use casts like this any time it is operating against a view. An even better fix would be to detect which columns in a view come from a table and which ones might not, and only use casts for the columns that aren't definitely from a table. The trick I was exploring here might be able to help with that: - #1293 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply 1174655187 | |
1074470568 | https://github.com/simonw/datasette/issues/1671#issuecomment-1074470568 | https://api.github.com/repos/simonw/datasette/issues/1671 | IC_kwDOBm6k_c5ACx6o | simonw 9599 | 2022-03-21T22:11:14Z | 2022-03-21T22:12:49Z | OWNER | I wonder if this will be a problem with generated columns, or with SQLite strict tables? My hunch is that strict tables will continue to work without any changes, because https://www.sqlite.org/stricttables.html says nothing about their impact on comparison operations. I should test this to make absolutely sure though. Generated columns have a type, so my hunch is they will continue to work fine too. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply 1174655187 | |
1074468450 | https://github.com/simonw/datasette/issues/1671#issuecomment-1074468450 | https://api.github.com/repos/simonw/datasette/issues/1671 | IC_kwDOBm6k_c5ACxZi | simonw 9599 | 2022-03-21T22:08:35Z | 2022-03-21T22:10:00Z | OWNER | Relevant section of the SQLite documentation: 3.2. Affinity Of Expressions:
In your example, Then 4.2. Type Conversions Prior To Comparison fills in the rest:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply 1174655187 | |
1074465536 | https://github.com/simonw/datasette/issues/1671#issuecomment-1074465536 | https://api.github.com/repos/simonw/datasette/issues/1671 | IC_kwDOBm6k_c5ACwsA | simonw 9599 | 2022-03-21T22:04:31Z | 2022-03-21T22:04:31Z | OWNER | Oh this is fascinating! I replicated the bug (thanks for the steps to reproduce) and it looks like this is down to the following: Against views, This doesn't happen against tables because of SQLite's type affinity mechanism, which handles the type conversion automatically. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply 1174655187 | |
1074459746 | https://github.com/simonw/datasette/issues/1679#issuecomment-1074459746 | https://api.github.com/repos/simonw/datasette/issues/1679 | IC_kwDOBm6k_c5ACvRi | simonw 9599 | 2022-03-21T21:55:45Z | 2022-03-21T21:55:45Z | OWNER | I'm going to change the original logic to set n=1 for times that are |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: how much overhead does the n=1 time limit have? 1175854982 | |
1074458506 | https://github.com/simonw/datasette/issues/1679#issuecomment-1074458506 | https://api.github.com/repos/simonw/datasette/issues/1679 | IC_kwDOBm6k_c5ACu-K | simonw 9599 | 2022-03-21T21:53:47Z | 2022-03-21T21:53:47Z | OWNER | Oh interesting, it turns out there is ONE place in the code that sets the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: how much overhead does the n=1 time limit have? 1175854982 | |
1074454687 | https://github.com/simonw/datasette/issues/1679#issuecomment-1074454687 | https://api.github.com/repos/simonw/datasette/issues/1679 | IC_kwDOBm6k_c5ACuCf | simonw 9599 | 2022-03-21T21:48:02Z | 2022-03-21T21:48:02Z | OWNER | Here's another microbenchmark that measures how many nanoseconds it takes to run 1,000 vmops: ```python import sqlite3 import time db = sqlite3.connect(":memory:") i = 0 out = [] def count(): global i i += 1000 out.append(((i, time.perf_counter_ns()))) db.set_progress_handler(count, 1000) print("Start:", time.perf_counter_ns()) all = db.execute(""" with recursive counter(x) as ( select 0 union select x + 1 from counter ) select * from counter limit 10000; """).fetchall() print("End:", time.perf_counter_ns()) print() print("So how long does it take to execute 1000 ops?") prev_time_ns = None
for i, time_ns in out:
if prev_time_ns is not None:
print(time_ns - prev_time_ns, "ns")
prev_time_ns = time_ns
So how long does it take to execute 1000 ops? 47290 ns 49573 ns 48226 ns 45674 ns 53238 ns 47313 ns 52346 ns 48689 ns 47092 ns 87596 ns 69999 ns 52522 ns 52809 ns 53259 ns 52478 ns 53478 ns 65812 ns ``` 87596ns is 0.087596ms - so even a measure rate of every 1000 ops is easily finely grained enough to capture differences of less than 0.1ms. If anything I could bump that default 1000 up - and I can definitely eliminate the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: how much overhead does the n=1 time limit have? 1175854982 | |
1074446576 | https://github.com/simonw/datasette/issues/1679#issuecomment-1074446576 | https://api.github.com/repos/simonw/datasette/issues/1679 | IC_kwDOBm6k_c5ACsDw | simonw 9599 | 2022-03-21T21:38:27Z | 2022-03-21T21:38:27Z | OWNER | OK here's a microbenchmark script: ```python import sqlite3 import timeit db = sqlite3.connect(":memory:") db_with_progress_handler_1 = sqlite3.connect(":memory:") db_with_progress_handler_1000 = sqlite3.connect(":memory:") db_with_progress_handler_1.set_progress_handler(lambda: None, 1) db_with_progress_handler_1000.set_progress_handler(lambda: None, 1000) def execute_query(db): cursor = db.execute(""" with recursive counter(x) as ( select 0 union select x + 1 from counter ) select * from counter limit 10000; """) list(cursor.fetchall()) print("Without progress_handler") print(timeit.timeit(lambda: execute_query(db), number=100)) print("progress_handler every 1000 ops") print(timeit.timeit(lambda: execute_query(db_with_progress_handler_1000), number=100)) print("progress_handler every 1 op")
print(timeit.timeit(lambda: execute_query(db_with_progress_handler_1), number=100))
So running every 1000 ops makes almost no difference at all, but running every single op is a 3.2x performance degradation. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: how much overhead does the n=1 time limit have? 1175854982 | |
1074439309 | https://github.com/simonw/datasette/issues/1679#issuecomment-1074439309 | https://api.github.com/repos/simonw/datasette/issues/1679 | IC_kwDOBm6k_c5ACqSN | simonw 9599 | 2022-03-21T21:28:58Z | 2022-03-21T21:28:58Z | OWNER | David Raymond solved it there: https://sqlite.org/forum/forumpost/330c8532d8a88bcd
Sure enough, adding that gets the VM steps number up to 190,007 which is close enough that I'm happy. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: how much overhead does the n=1 time limit have? 1175854982 | |
1074378472 | https://github.com/simonw/datasette/issues/1676#issuecomment-1074378472 | https://api.github.com/repos/simonw/datasette/issues/1676 | IC_kwDOBm6k_c5ACbbo | simonw 9599 | 2022-03-21T20:18:10Z | 2022-03-21T20:18:10Z | OWNER | Maybe there is a better name for this method that helps emphasize its cascading nature. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Reconsider ensure_permissions() logic, can it be less confusing? 1175690070 | |
1074347023 | https://github.com/simonw/datasette/issues/1679#issuecomment-1074347023 | https://api.github.com/repos/simonw/datasette/issues/1679 | IC_kwDOBm6k_c5ACTwP | simonw 9599 | 2022-03-21T19:48:59Z | 2022-03-21T19:48:59Z | OWNER | Posed a question about that here: https://sqlite.org/forum/forumpost/de9ff10fa7 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: how much overhead does the n=1 time limit have? 1175854982 | |
1074341924 | https://github.com/simonw/datasette/issues/1679#issuecomment-1074341924 | https://api.github.com/repos/simonw/datasette/issues/1679 | IC_kwDOBm6k_c5ACSgk | simonw 9599 | 2022-03-21T19:42:08Z | 2022-03-21T19:42:08Z | OWNER | Here's the Python-C implementation of It calls https://www.sqlite.org/c3ref/progress_handler.html says:
So maybe VM-steps and virtual machine instructions are different things? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: how much overhead does the n=1 time limit have? 1175854982 | |
1074337997 | https://github.com/simonw/datasette/issues/1679#issuecomment-1074337997 | https://api.github.com/repos/simonw/datasette/issues/1679 | IC_kwDOBm6k_c5ACRjN | simonw 9599 | 2022-03-21T19:37:08Z | 2022-03-21T19:37:08Z | OWNER | This is weird: ```python import sqlite3 db = sqlite3.connect(":memory:") i = 0 def count(): global i i += 1 db.set_progress_handler(count, 1) db.execute(""" with recursive counter(x) as ( select 0 union select x + 1 from counter ) select * from counter limit 10000; """) print(i)
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: how much overhead does the n=1 time limit have? 1175854982 | |
1074332718 | https://github.com/simonw/datasette/issues/1679#issuecomment-1074332718 | https://api.github.com/repos/simonw/datasette/issues/1679 | IC_kwDOBm6k_c5ACQQu | simonw 9599 | 2022-03-21T19:31:10Z | 2022-03-21T19:31:10Z | OWNER | How long does it take for SQLite to execute 1000 opcodes anyway? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: how much overhead does the n=1 time limit have? 1175854982 | |
1074332325 | https://github.com/simonw/datasette/issues/1679#issuecomment-1074332325 | https://api.github.com/repos/simonw/datasette/issues/1679 | IC_kwDOBm6k_c5ACQKl | simonw 9599 | 2022-03-21T19:30:44Z | 2022-03-21T19:30:44Z | OWNER | So it looks like even for facet suggestion |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: how much overhead does the n=1 time limit have? 1175854982 | |
1074331743 | https://github.com/simonw/datasette/issues/1679#issuecomment-1074331743 | https://api.github.com/repos/simonw/datasette/issues/1679 | IC_kwDOBm6k_c5ACQBf | simonw 9599 | 2022-03-21T19:30:05Z | 2022-03-21T19:30:05Z | OWNER | https://github.com/simonw/datasette/blob/1a7750eb29fd15dd2eea3b9f6e33028ce441b143/datasette/app.py#L118-L122 sets it to 50ms for facet suggestion but that's not going to pass
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: how much overhead does the n=1 time limit have? 1175854982 | |
1074321862 | https://github.com/simonw/datasette/issues/1660#issuecomment-1074321862 | https://api.github.com/repos/simonw/datasette/issues/1660 | IC_kwDOBm6k_c5ACNnG | simonw 9599 | 2022-03-21T19:19:01Z | 2022-03-21T19:19:01Z | OWNER | I've simplified this a ton now. I'm going to keep working on this in the long-term but I think this issue can be closed. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor and simplify Datasette routing and views 1170144879 | |
1074302559 | https://github.com/simonw/datasette/issues/1678#issuecomment-1074302559 | https://api.github.com/repos/simonw/datasette/issues/1678 | IC_kwDOBm6k_c5ACI5f | simonw 9599 | 2022-03-21T19:04:03Z | 2022-03-21T19:04:03Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Make `check_visibility()` a documented API 1175715988 | ||
1074287177 | https://github.com/simonw/datasette/issues/1660#issuecomment-1074287177 | https://api.github.com/repos/simonw/datasette/issues/1660 | IC_kwDOBm6k_c5ACFJJ | simonw 9599 | 2022-03-21T18:51:42Z | 2022-03-21T18:51:42Z | OWNER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor and simplify Datasette routing and views 1170144879 | |
1074256603 | https://github.com/simonw/sqlite-utils/issues/417#issuecomment-1074256603 | https://api.github.com/repos/simonw/sqlite-utils/issues/417 | IC_kwDOCGYnMM5AB9rb | blaine 9954 | 2022-03-21T18:19:41Z | 2022-03-21T18:19:41Z | NONE | That makes sense; just a little hint that points folks towards doing the right thing might be helpful! fwiw, the reason I was using jq in the first place was just a quick way to extract one attribute from an actual JSON array. When I initially imported it, I got a table with a bunch of embedded JSON values, rather than a native table, because each array entry had two attributes, one with the data I actually wanted. Not sure how common a use-case this is, though (and easily fixed, aside from the jq weirdness!) |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
insert fails on JSONL with whitespace 1175744654 | |
1074243540 | https://github.com/simonw/sqlite-utils/issues/417#issuecomment-1074243540 | https://api.github.com/repos/simonw/sqlite-utils/issues/417 | IC_kwDOCGYnMM5AB6fU | simonw 9599 | 2022-03-21T18:08:03Z | 2022-03-21T18:08:03Z | OWNER | I've not really thought about standards as much here as I should. It looks like there are two competing specs for newline-delimited JSON! http://ndjson.org/ is the one I've been using in
https://jsonlines.org/ is the other one. It is slightly less clear, but it does say this:
My interpretation of both of these is that newlines in the middle of a JSON object shouldn't be allowed. So what's The The thing I like about newline-delimited JSON is that it's really trivial to parse - loop through each line, run it through Unless someone has written a robust Python implementation of a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
insert fails on JSONL with whitespace 1175744654 | |
1074184240 | https://github.com/simonw/datasette/issues/1677#issuecomment-1074184240 | https://api.github.com/repos/simonw/datasette/issues/1677 | IC_kwDOBm6k_c5ABsAw | simonw 9599 | 2022-03-21T17:20:17Z | 2022-03-21T17:20:17Z | OWNER | This is weirdly different from how |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Remove `check_permission()` from `BaseView` 1175694248 | |
1074180312 | https://github.com/simonw/datasette/issues/1676#issuecomment-1074180312 | https://api.github.com/repos/simonw/datasette/issues/1676 | IC_kwDOBm6k_c5ABrDY | simonw 9599 | 2022-03-21T17:16:45Z | 2022-03-21T17:16:45Z | OWNER | When looking at this code earlier I assumed that the following would check each permission in turn and fail if any of them failed:
If that is indeed the right abstraction, I need to work to make the documentation as clear as possible. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Reconsider ensure_permissions() logic, can it be less confusing? 1175690070 | |
1074178865 | https://github.com/simonw/datasette/issues/1676#issuecomment-1074178865 | https://api.github.com/repos/simonw/datasette/issues/1676 | IC_kwDOBm6k_c5ABqsx | simonw 9599 | 2022-03-21T17:15:27Z | 2022-03-21T17:15:27Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Reconsider ensure_permissions() logic, can it be less confusing? 1175690070 | ||
1074177827 | https://github.com/simonw/datasette/issues/1675#issuecomment-1074177827 | https://api.github.com/repos/simonw/datasette/issues/1675 | IC_kwDOBm6k_c5ABqcj | simonw 9599 | 2022-03-21T17:14:31Z | 2022-03-21T17:14:31Z | OWNER | Updated documentation: https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/docs/internals.rst#await-ensure_permissionsactor-permissions
That's pretty hard to understand! I'm going to open a separate issue to reconsider if this is a useful enough abstraction given how confusing it is. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Extract out `check_permissions()` from `BaseView 1175648453 | |
1074161523 | https://github.com/simonw/datasette/issues/1675#issuecomment-1074161523 | https://api.github.com/repos/simonw/datasette/issues/1675 | IC_kwDOBm6k_c5ABmdz | simonw 9599 | 2022-03-21T16:59:55Z | 2022-03-21T17:00:03Z | OWNER | Also calling that function |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Extract out `check_permissions()` from `BaseView 1175648453 | |
1074158890 | https://github.com/simonw/datasette/issues/1675#issuecomment-1074158890 | https://api.github.com/repos/simonw/datasette/issues/1675 | IC_kwDOBm6k_c5ABl0q | simonw 9599 | 2022-03-21T16:57:15Z | 2022-03-21T16:57:15Z | OWNER | Idea: A new |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Extract out `check_permissions()` from `BaseView 1175648453 | |
1074156779 | https://github.com/simonw/datasette/issues/1675#issuecomment-1074156779 | https://api.github.com/repos/simonw/datasette/issues/1675 | IC_kwDOBm6k_c5ABlTr | simonw 9599 | 2022-03-21T16:55:08Z | 2022-03-21T16:56:02Z | OWNER | One benefit of the current design of I could return an object which evaluates to |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Extract out `check_permissions()` from `BaseView 1175648453 | |
1074143209 | https://github.com/simonw/datasette/issues/1675#issuecomment-1074143209 | https://api.github.com/repos/simonw/datasette/issues/1675 | IC_kwDOBm6k_c5ABh_p | simonw 9599 | 2022-03-21T16:46:05Z | 2022-03-21T16:46:05Z | OWNER | The other difference though is that |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Extract out `check_permissions()` from `BaseView 1175648453 | |
1074142617 | https://github.com/simonw/datasette/issues/1675#issuecomment-1074142617 | https://api.github.com/repos/simonw/datasette/issues/1675 | IC_kwDOBm6k_c5ABh2Z | simonw 9599 | 2022-03-21T16:45:27Z | 2022-03-21T16:45:27Z | OWNER | Though at that point So maybe
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Extract out `check_permissions()` from `BaseView 1175648453 | |
1074141457 | https://github.com/simonw/datasette/issues/1675#issuecomment-1074141457 | https://api.github.com/repos/simonw/datasette/issues/1675 | IC_kwDOBm6k_c5ABhkR | simonw 9599 | 2022-03-21T16:44:09Z | 2022-03-21T16:44:09Z | OWNER | A slightly odd thing about these methods is that they either fail silently or they raise a Maybe they should instead return |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Extract out `check_permissions()` from `BaseView 1175648453 | |
1074136176 | https://github.com/simonw/datasette/issues/1660#issuecomment-1074136176 | https://api.github.com/repos/simonw/datasette/issues/1660 | IC_kwDOBm6k_c5ABgRw | simonw 9599 | 2022-03-21T16:38:46Z | 2022-03-21T16:38:46Z | OWNER | I'm going to refactor this stuff out and document it so it can be easily used by plugins: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor and simplify Datasette routing and views 1170144879 | |
1074019047 | https://github.com/simonw/datasette/issues/526#issuecomment-1074019047 | https://api.github.com/repos/simonw/datasette/issues/526 | IC_kwDOBm6k_c5ABDrn | simonw 9599 | 2022-03-21T15:09:56Z | 2022-03-21T15:09:56Z | OWNER | I should research how much overhead creating a new connection costs - it may be that an easy way to solve this is to create A dedicated connection for the query and then close that connection at the end. |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Stream all results for arbitrary SQL and canned queries 459882902 | |
1074017633 | https://github.com/simonw/datasette/issues/1177#issuecomment-1074017633 | https://api.github.com/repos/simonw/datasette/issues/1177 | IC_kwDOBm6k_c5ABDVh | simonw 9599 | 2022-03-21T15:08:51Z | 2022-03-21T15:08:51Z | OWNER | Related: - #1062 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Ability to stream all rows as newline-delimited JSON 780153562 | |
1073468996 | https://github.com/simonw/sqlite-utils/issues/415#issuecomment-1073468996 | https://api.github.com/repos/simonw/sqlite-utils/issues/415 | IC_kwDOCGYnMM4_-9ZE | simonw 9599 | 2022-03-21T04:14:42Z | 2022-03-21T04:14:42Z | OWNER | I can fix this like so: ``` % sqlite-utils convert demo.db demo foo '{"foo": "bar"}' --multi --dry-run abc --- becomes: {"foo": "bar"} Would affect 1 row
|
{ "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Convert with `--multi` and `--dry-run` flag does not work 1171599874 | |
1073463375 | https://github.com/simonw/sqlite-utils/issues/415#issuecomment-1073463375 | https://api.github.com/repos/simonw/sqlite-utils/issues/415 | IC_kwDOCGYnMM4_-8BP | simonw 9599 | 2022-03-21T04:02:36Z | 2022-03-21T04:02:36Z | OWNER | Thanks for the really clear steps to reproduce! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Convert with `--multi` and `--dry-run` flag does not work 1171599874 | |
1073456222 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073456222 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_-6Re | simonw 9599 | 2022-03-21T03:45:52Z | 2022-03-21T03:45:52Z | OWNER | Needs tests and documentation. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
1073456155 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073456155 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_-6Qb | simonw 9599 | 2022-03-21T03:45:37Z | 2022-03-21T03:45:37Z | OWNER | Prototype: ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index 8255b56..0a3693e 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -2583,7 +2583,11 @@ def generate_convert_help(): """ ).strip() recipe_names = [ - n for n in dir(recipes) if not n.startswith("") and n not in ("json", "parser") + n + for n in dir(recipes) + if not n.startswith("_") + and n not in ("json", "parser") + and callable(getattr(recipes, n)) ] for name in recipe_names: fn = getattr(recipes, name) diff --git a/sqlite_utils/recipes.py b/sqlite_utils/recipes.py index 6918661..569c30d 100644 --- a/sqlite_utils/recipes.py +++ b/sqlite_utils/recipes.py @@ -1,17 +1,38 @@ from dateutil import parser import json +IGNORE = object() +SET_NULL = object() -def parsedate(value, dayfirst=False, yearfirst=False): + +def parsedate(value, dayfirst=False, yearfirst=False, errors=None): "Parse a date and convert it to ISO date format: yyyy-mm-dd" - return ( - parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).date().isoformat() - ) + try: + return ( + parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst) + .date() + .isoformat() + ) + except parser.ParserError: + if errors is IGNORE: + return value + elif errors is SET_NULL: + return None + else: + raise -def parsedatetime(value, dayfirst=False, yearfirst=False): +def parsedatetime(value, dayfirst=False, yearfirst=False, errors=None): "Parse a datetime and convert it to ISO datetime format: yyyy-mm-ddTHH:MM:SS" - return parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).isoformat() + try: + return parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).isoformat() + except parser.ParserError: + if errors is IGNORE: + return value + elif errors is SET_NULL: + return None + else: + raise def jsonsplit(value, delimiter=",", type=str): ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
1073455905 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073455905 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_-6Mh | simonw 9599 | 2022-03-21T03:44:47Z | 2022-03-21T03:45:00Z | OWNER | This is quite nice:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
1073453370 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073453370 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_-5k6 | simonw 9599 | 2022-03-21T03:41:06Z | 2022-03-21T03:41:06Z | OWNER | I'm going to try the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
1073453230 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073453230 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_-5iu | simonw 9599 | 2022-03-21T03:40:37Z | 2022-03-21T03:40:37Z | OWNER | I think the options here should be:
These need to be indicated by parameters to the Some design options:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
1073451659 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073451659 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_-5KL | simonw 9599 | 2022-03-21T03:35:01Z | 2022-03-21T03:35:01Z | OWNER | I confirmed that if it fails for any value ALL values are left alone, since it runs in a transaction. Here's the code that does that: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
1073450588 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073450588 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_-45c | simonw 9599 | 2022-03-21T03:32:58Z | 2022-03-21T03:32:58Z | OWNER | Then I ran this to convert
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
1073448904 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073448904 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_-4fI | simonw 9599 | 2022-03-21T03:28:12Z | 2022-03-21T03:30:37Z | OWNER | Generating a test database using a pattern from https://www.geekytidbits.com/date-range-table-sqlite/
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
1073366630 | https://github.com/simonw/datasette/issues/1510#issuecomment-1073366630 | https://api.github.com/repos/simonw/datasette/issues/1510 | IC_kwDOBm6k_c4_-kZm | simonw 9599 | 2022-03-20T22:59:33Z | 2022-03-20T22:59:33Z | OWNER | I really like the idea of making this effectively the same thing as the fully documented, stable JSON API that comes as part of 1.0. If you want to know what will be available to your templates, consult the API documentation. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Datasette 1.0 documented template context (maybe via API docs) 1054244712 | |
1073366436 | https://github.com/simonw/datasette/issues/1674#issuecomment-1073366436 | https://api.github.com/repos/simonw/datasette/issues/1674 | IC_kwDOBm6k_c4_-kWk | simonw 9599 | 2022-03-20T22:58:40Z | 2022-03-20T22:58:40Z | OWNER | This will probably happen as part of turning this into an officially documented API that serves the template context for the homepage: - #1510 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Tweak design of /.json 1174717287 | |
1073362979 | https://github.com/simonw/datasette/issues/1355#issuecomment-1073362979 | https://api.github.com/repos/simonw/datasette/issues/1355 | IC_kwDOBm6k_c4_-jgj | simonw 9599 | 2022-03-20T22:38:53Z | 2022-03-20T22:38:53Z | OWNER | Built a research prototype: ```diff diff --git a/datasette/app.py b/datasette/app.py index 5c8101a..5cd3e63 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -1,6 +1,7 @@ import asyncio import asgi_csrf import collections +import contextlib import datetime import functools import glob @@ -1490,3 +1491,11 @@ class DatasetteClient: return await client.request( method, self._fix(path, avoid_path_rewrites), kwargs ) + + @contextlib.asynccontextmanager + async def stream(self, method, path, kwargs): + async with httpx.AsyncClient(app=self.app) as client: + print("async with as client") + async with client.stream(method, self._fix(path), **kwargs) as response: + print("async with client.stream about to yield response") + yield response diff --git a/datasette/cli.py b/datasette/cli.py index 3c6e1b2..3025ead 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -585,11 +585,19 @@ def serve( asyncio.get_event_loop().run_until_complete(check_databases(ds))
No idea why. I dropped this for the moment. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette --get should efficiently handle streaming CSV 910088936 | |
1073361986 | https://github.com/simonw/datasette/issues/1673#issuecomment-1073361986 | https://api.github.com/repos/simonw/datasette/issues/1673 | IC_kwDOBm6k_c4_-jRC | simonw 9599 | 2022-03-20T22:31:41Z | 2022-03-20T22:34:06Z | OWNER | Maybe it's because Actually no, I'm caching that already: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Streaming CSV spends a lot of time in `table_column_details` 1174708375 | |
1073355818 | https://github.com/simonw/datasette/issues/1672#issuecomment-1073355818 | https://api.github.com/repos/simonw/datasette/issues/1672 | IC_kwDOBm6k_c4_-hwq | simonw 9599 | 2022-03-20T21:52:38Z | 2022-03-20T21:52:38Z | OWNER | That means taking on these issues: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor CSV handling code out of DataView 1174697144 | |
1073355032 | https://github.com/simonw/datasette/issues/1660#issuecomment-1073355032 | https://api.github.com/repos/simonw/datasette/issues/1660 | IC_kwDOBm6k_c4_-hkY | simonw 9599 | 2022-03-20T21:46:43Z | 2022-03-20T21:46:43Z | OWNER | I think the way to get rid of most of the remaining complexity in |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor and simplify Datasette routing and views 1170144879 | |
1073330388 | https://github.com/simonw/sqlite-utils/issues/140#issuecomment-1073330388 | https://api.github.com/repos/simonw/sqlite-utils/issues/140 | IC_kwDOCGYnMM4_-bjU | simonw 9599 | 2022-03-20T19:44:39Z | 2022-03-20T19:45:45Z | OWNER | Alternative idea for specifying types: accept a Python expression, then use Python type literal syntax. For example:
Where |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Idea: insert-files mechanism for adding extra columns with fixed values 688351054 | |
1073152522 | https://github.com/dogsheep/google-takeout-to-sqlite/issues/10#issuecomment-1073152522 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/10 | IC_kwDODFE5qs4_9wIK | csusanu 9290214 | 2022-03-20T02:38:07Z | 2022-03-20T02:38:07Z | NONE | This line needs to say |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite3.OperationalError: no such table: main.my_activity 1123393829 | |
1073143413 | https://github.com/simonw/datasette/issues/1669#issuecomment-1073143413 | https://api.github.com/repos/simonw/datasette/issues/1669 | IC_kwDOBm6k_c4_9t51 | simonw 9599 | 2022-03-20T01:24:36Z | 2022-03-20T01:24:36Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Release 0.61 alpha 1174404647 | ||
1073139067 | https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-1073139067 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14 | IC_kwDOC8tyDs4_9s17 | lchski 343884 | 2022-03-20T00:54:18Z | 2022-03-20T00:54:18Z | NONE | Update: this appears to be because of running the command twice without clearing the DB in between. Tries to insert a Workout that already exists, causing a collision on the (auto-generated) |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
UNIQUE constraint failed: workouts.id 771608692 | |
1073137170 | https://github.com/simonw/datasette/issues/1669#issuecomment-1073137170 | https://api.github.com/repos/simonw/datasette/issues/1669 | IC_kwDOBm6k_c4_9sYS | simonw 9599 | 2022-03-20T00:35:52Z | 2022-03-20T00:35:52Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Release 0.61 alpha 1174404647 | ||
1073136896 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073136896 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9sUA | simonw 9599 | 2022-03-20T00:33:23Z | 2022-03-20T00:33:23Z | OWNER | I'm going to release this as a 0.61 alpha so I can more easily depend on it from |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introduce concept of a database `route`, separate from its name 1174306154 | |
1073136686 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073136686 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9sQu | simonw 9599 | 2022-03-20T00:31:13Z | 2022-03-20T00:31:13Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introduce concept of a database `route`, separate from its name 1174306154 | ||
1073135433 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073135433 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9r9J | simonw 9599 | 2022-03-20T00:20:36Z | 2022-03-20T00:20:36Z | OWNER | Building this plugin instantly revealed that all of the links - on the homepage and the database page and so on - are incorrect: ```python from datasette import hookimpl @hookimpl def startup(datasette): db = datasette.get_database("fixtures2") db.route = "alternative-route" ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introduce concept of a database `route`, separate from its name 1174306154 | |
1073134816 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073134816 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9rzg | simonw 9599 | 2022-03-20T00:16:22Z | 2022-03-20T00:16:22Z | OWNER | I'm going to add a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introduce concept of a database `route`, separate from its name 1174306154 | |
1073134206 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073134206 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9rp- | simonw 9599 | 2022-03-20T00:12:03Z | 2022-03-20T00:12:03Z | OWNER | I'd like to have a live demo of this up on |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introduce concept of a database `route`, separate from its name 1174306154 | |
1073126264 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073126264 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9pt4 | simonw 9599 | 2022-03-19T22:59:30Z | 2022-03-19T22:59:30Z | OWNER | Also need to update the Need to add tests that check the links in the HTML and can confirm this is working correctly. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introduce concept of a database `route`, separate from its name 1174306154 | |
1073125334 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073125334 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9pfW | simonw 9599 | 2022-03-19T22:53:55Z | 2022-03-19T22:53:55Z | OWNER | Need to update documentation in a few places - e.g. https://docs.datasette.io/en/stable/internals.html#remove-database-name
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introduce concept of a database `route`, separate from its name 1174306154 | |
1073123231 | https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-1073123231 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14 | IC_kwDOC8tyDs4_9o-f | lchski 343884 | 2022-03-19T22:39:29Z | 2022-03-19T22:39:29Z | NONE | I have this issue, too, with a fresh export. None of my When I run the script, a
Are there maybe duplicate workouts in the data, which’d cause multiple rows to share the same |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
UNIQUE constraint failed: workouts.id 771608692 | |
1073112104 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073112104 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9mQo | simonw 9599 | 2022-03-19T21:08:21Z | 2022-03-19T21:08:21Z | OWNER | I think I've got this working but I need to write a test for it that covers the rare case when the route is not the same thing as the database name. I'll do that with a new test. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introduce concept of a database `route`, separate from its name 1174306154 | |
1073097394 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073097394 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9iqy | simonw 9599 | 2022-03-19T20:56:35Z | 2022-03-19T20:56:35Z | OWNER | I'm trying to think if there's any reason not to use Decision made: I'm going with |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introduce concept of a database `route`, separate from its name 1174306154 | |
1073076624 | https://github.com/simonw/datasette/issues/1667#issuecomment-1073076624 | https://api.github.com/repos/simonw/datasette/issues/1667 | IC_kwDOBm6k_c4_9dmQ | simonw 9599 | 2022-03-19T20:31:44Z | 2022-03-19T20:31:44Z | OWNER | I can now read |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Make route matched pattern groups more consistent 1174302994 | |
1073076187 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073076187 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9dfb | simonw 9599 | 2022-03-19T20:28:20Z | 2022-03-19T20:28:20Z | OWNER | I'm going to keep |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introduce concept of a database `route`, separate from its name 1174306154 | |
1073076136 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073076136 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9deo | simonw 9599 | 2022-03-19T20:27:44Z | 2022-03-19T20:27:44Z | OWNER | Pretty sure changing it will break some existing plugins though, including likely Datasette Desktop. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introduce concept of a database `route`, separate from its name 1174306154 | |
1073076110 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073076110 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9deO | simonw 9599 | 2022-03-19T20:27:22Z | 2022-03-19T20:27:22Z | OWNER | The docs do currently describe Good thing I'm not at 1.0 yet so I can change that! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introduce concept of a database `route`, separate from its name 1174306154 | |
1073076015 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073076015 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9dcv | simonw 9599 | 2022-03-19T20:26:32Z | 2022-03-19T20:26:32Z | OWNER | I'm inclined to redefine |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introduce concept of a database `route`, separate from its name 1174306154 | |
1073075913 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073075913 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9dbJ | simonw 9599 | 2022-03-19T20:25:46Z | 2022-03-19T20:26:08Z | OWNER | The output of
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introduce concept of a database `route`, separate from its name 1174306154 | |
1073075697 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073075697 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9dXx | simonw 9599 | 2022-03-19T20:24:06Z | 2022-03-19T20:24:06Z | OWNER | Right now if a database has a
But the output on |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introduce concept of a database `route`, separate from its name 1174306154 | |
1073073599 | https://github.com/simonw/datasette/issues/1660#issuecomment-1073073599 | https://api.github.com/repos/simonw/datasette/issues/1660 | IC_kwDOBm6k_c4_9c2_ | simonw 9599 | 2022-03-19T20:06:40Z | 2022-03-19T20:06:40Z | OWNER | This blocks: - #1668 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor and simplify Datasette routing and views 1170144879 | |
1073073579 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073073579 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9c2r | simonw 9599 | 2022-03-19T20:06:27Z | 2022-03-19T20:06:27Z | OWNER | Marking this as blocked until #1660 is done. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introduce concept of a database `route`, separate from its name 1174306154 | |
1073073547 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073073547 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9c2L | simonw 9599 | 2022-03-19T20:06:07Z | 2022-03-19T20:06:07Z | OWNER | Implementing this is a little tricky because there's a whole lot of code that expects the The All the more reason to get rid of that |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introduce concept of a database `route`, separate from its name 1174306154 | |
1073043433 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073043433 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9Vfp | simonw 9599 | 2022-03-19T16:54:55Z | 2022-03-19T20:01:19Z | OWNER | Options:
- I like |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introduce concept of a database `route`, separate from its name 1174306154 | |
1073043713 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073043713 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9VkB | simonw 9599 | 2022-03-19T16:56:19Z | 2022-03-19T16:56:19Z | OWNER | Worth noting that the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introduce concept of a database `route`, separate from its name 1174306154 | |
1073043350 | https://github.com/simonw/datasette/issues/1668#issuecomment-1073043350 | https://api.github.com/repos/simonw/datasette/issues/1668 | IC_kwDOBm6k_c4_9VeW | simonw 9599 | 2022-03-19T16:54:26Z | 2022-03-19T16:54:26Z | OWNER | The So need a different name for the path-that-is-used-in-the-URL. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introduce concept of a database `route`, separate from its name 1174306154 | |
1073042554 | https://github.com/simonw/datasette/issues/1667#issuecomment-1073042554 | https://api.github.com/repos/simonw/datasette/issues/1667 | IC_kwDOBm6k_c4_9VR6 | simonw 9599 | 2022-03-19T16:50:01Z | 2022-03-19T16:52:35Z | OWNER | OK, I've made this more consistent - I still need to address the fact that |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Make route matched pattern groups more consistent 1174302994 | |
1073040072 | https://github.com/simonw/datasette/issues/1667#issuecomment-1073040072 | https://api.github.com/repos/simonw/datasette/issues/1667 | IC_kwDOBm6k_c4_9UrI | simonw 9599 | 2022-03-19T16:34:02Z | 2022-03-19T16:34:02Z | OWNER | I called it I think I'm going to go with |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Make route matched pattern groups more consistent 1174302994 | |
1073039670 | https://github.com/simonw/datasette/issues/1666#issuecomment-1073039670 | https://api.github.com/repos/simonw/datasette/issues/1666 | IC_kwDOBm6k_c4_9Uk2 | simonw 9599 | 2022-03-19T16:31:08Z | 2022-03-19T16:31:57Z | OWNER | This does make it more interesting - it also highlights how inconsistent the way the capturing works is. Especially |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor URL routing to enable testing 1174162781 | |
1073039241 | https://github.com/simonw/datasette/issues/1666#issuecomment-1073039241 | https://api.github.com/repos/simonw/datasette/issues/1666 | IC_kwDOBm6k_c4_9UeJ | simonw 9599 | 2022-03-19T16:28:15Z | 2022-03-19T16:28:15Z | OWNER | This is more interesting if it also asserts against the captured matches from the pattern. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor URL routing to enable testing 1174162781 | |
1073037939 | https://github.com/simonw/datasette/issues/878#issuecomment-1073037939 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c4_9UJz | simonw 9599 | 2022-03-19T16:19:30Z | 2022-03-19T16:19:30Z | OWNER | On revisiting https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2 a few months later I'm having second thoughts about using But I still like the pattern as a way to resolve more complex cases like "to generate GeoJSON of the expanded view with labels, the label expansion code needs to run once at some before the GeoJSON formatting code does". So I'm going to stick with it a tiny bit longer, but maybe try to make it a lot more explicit when it's going to happen rather than having the main view methods themselves also use async DI. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
1072954795 | https://github.com/simonw/datasette/issues/1228#issuecomment-1072954795 | https://api.github.com/repos/simonw/datasette/issues/1228 | IC_kwDOBm6k_c4_8_2r | Kabouik 7107523 | 2022-03-19T06:44:40Z | 2022-03-19T06:44:40Z | NONE |
Exactly, that's highly likely even though I can't double check from this computer just now. Thanks! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
500 error caused by faceting if a column called `n` exists 810397025 | |
1072939780 | https://github.com/simonw/datasette/issues/1561#issuecomment-1072939780 | https://api.github.com/repos/simonw/datasette/issues/1561 | IC_kwDOBm6k_c4_88ME | simonw 9599 | 2022-03-19T04:45:40Z | 2022-03-19T04:45:40Z | OWNER | I ended up moving hashed URL mode out to a plugin in: - #647 If you're still interested in using it with |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
add hash id to "_memory" url if hashed url mode is turned on and crossdb is also turned on 1082765654 | |
1072933875 | https://github.com/simonw/datasette/issues/1666#issuecomment-1072933875 | https://api.github.com/repos/simonw/datasette/issues/1666 | IC_kwDOBm6k_c4_86vz | simonw 9599 | 2022-03-19T04:03:42Z | 2022-03-19T04:03:42Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor URL routing to enable testing 1174162781 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
user >30