issue_comments

Table actions

6,390 rows sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

issue >1000

user 232

author_association 4

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
927312650 https://github.com/dogsheep/twitter-to-sqlite/issues/54#issuecomment-927312650 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54 IC_kwDODEm0Qs43RasK danp 2182 2021-09-26T14:09:51Z 2021-09-26T14:09:51Z NONE

Similar trouble with ageinfo using 0.22. Here's what my ageinfo.js file looks like:

window.YTD.ageinfo.part0 = [
  {
    "ageMeta" : { }
  }
]

Commenting out the registration for ageinfo in archive.py gets my archive to import.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Archive import appears to be broken on recent exports 779088071  
926208819 https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926208819 https://api.github.com/repos/simonw/sqlite-utils/issues/329 IC_kwDOCGYnMM43NNMz simonw 9599 2021-09-23T22:26:54Z 2021-09-23T22:26:54Z OWNER

I could even have those replacement characters be properties of the Database class, so uses can sub-class and change them.

class Database:
    left_brace_replace = "_"
    right_brace_replace = "_"
    ...

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Rethink approach to [ and ] in column names (currently throws error) 1005891028  
926207719 https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926207719 https://api.github.com/repos/simonw/sqlite-utils/issues/329 IC_kwDOCGYnMM43NM7n simonw 9599 2021-09-23T22:24:09Z 2021-09-23T22:24:09Z OWNER

I think I like the underscore option best. I don't like the idea of injecting surprise ( ) parenthesis, and having them vanish entirely could result in things like item[price] becoming itemprice which feels confusing.

item_price_ is a little ugly but I think I can live with it.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Rethink approach to [ and ] in column names (currently throws error) 1005891028  
926207246 https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926207246 https://api.github.com/repos/simonw/sqlite-utils/issues/329 IC_kwDOCGYnMM43NM0O simonw 9599 2021-09-23T22:23:09Z 2021-09-23T22:23:09Z OWNER

What are my options for replacing those characters?

  • [ becomes ( and ] becomes )
  • [ and ] are removed entirely
  • [ and ] both become _
 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Rethink approach to [ and ] in column names (currently throws error) 1005891028  
926206705 https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926206705 https://api.github.com/repos/simonw/sqlite-utils/issues/329 IC_kwDOCGYnMM43NMrx simonw 9599 2021-09-23T22:21:58Z 2021-09-23T22:21:58Z OWNER

I'm inclined to just fix them and not have an option for opting-out of fixing them, since it adds quite a bit of cruft to the overall API design for an option that maybe no-one will ever use.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Rethink approach to [ and ] in column names (currently throws error) 1005891028  
926206220 https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926206220 https://api.github.com/repos/simonw/sqlite-utils/issues/329 IC_kwDOCGYnMM43NMkM simonw 9599 2021-09-23T22:20:55Z 2021-09-23T22:21:19Z OWNER

If I add a new parameter for opting in and out of fixing these, what should it be called? A few options:

  • fix_columns=False
  • rename_bad_columns=False
  • fix_column_names=False
  • error_on_invalid_column_names=True
 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Rethink approach to [ and ] in column names (currently throws error) 1005891028  
926205047 https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926205047 https://api.github.com/repos/simonw/sqlite-utils/issues/329 IC_kwDOCGYnMM43NMR3 simonw 9599 2021-09-23T22:18:34Z 2021-09-23T22:19:38Z OWNER

Here's the code where this happens: https://github.com/simonw/sqlite-utils/blob/54191d4dc114d7dc21e849b48a4d5ae4f9e601ca/sqlite_utils/db.py#L2943-L2948

It's called from three different methods in db.py: create_table_sql(), update() and insert_all().

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Rethink approach to [ and ] in column names (currently throws error) 1005891028  
926204046 https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926204046 https://api.github.com/repos/simonw/sqlite-utils/issues/329 IC_kwDOCGYnMM43NMCO simonw 9599 2021-09-23T22:16:18Z 2021-09-23T22:16:18Z OWNER

So either I do the automatic replacement, or I let the user request automatic replacement, or a third option: I do automatic replacement but let the user opt to receive an error instead.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Rethink approach to [ and ] in column names (currently throws error) 1005891028  
926203501 https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926203501 https://api.github.com/repos/simonw/sqlite-utils/issues/329 IC_kwDOCGYnMM43NL5t simonw 9599 2021-09-23T22:15:07Z 2021-09-23T22:15:30Z OWNER

Quoting https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586676856 in full:

I'm not sure what to do about this one.

I can't fix it: this bug in Python's sqlite3 module means that even if I write a database out with column names that include [] I won't be able to read them back again.

So... I could do one of the following:

  • Throw an error if a column name includes those characters. That's my preferred option I think.
  • Automatically replace [ in column names with ( and ] with )
  • Do the automatic replacement but show a user-visible warning when I do it
  • Throw an error, but give the user an option to run with e.g. --fix-column-names which applies that automatic fix.

Since this is likely to be an incredibly rare edge-case I think I'd rather minimize the amount of code that deals with it, so my preferred option is to just throw that error and stop.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Rethink approach to [ and ] in column names (currently throws error) 1005891028  
925321439 https://github.com/simonw/sqlite-utils/issues/325#issuecomment-925321439 https://api.github.com/repos/simonw/sqlite-utils/issues/325 IC_kwDOCGYnMM43J0jf simonw 9599 2021-09-22T20:52:56Z 2021-09-22T20:52:56Z OWNER

Updated documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#running-queries-directly-against-csv-or-json

If two files have the same name they will be assigned a numeric suffix:

$ sqlite-utils memory foo/data.csv bar/data.csv "select * from data_2"

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 sqlite-utils memory can't deal with multiple files with the same name 990844088  
925303497 https://github.com/simonw/sqlite-utils/issues/325#issuecomment-925303497 https://api.github.com/repos/simonw/sqlite-utils/issues/325 IC_kwDOCGYnMM43JwLJ simonw 9599 2021-09-22T20:25:44Z 2021-09-22T20:25:44Z OWNER

Here's the relevant code: https://github.com/simonw/sqlite-utils/blob/7427a9137f60de961b6331d0922a3f03da0d1890/sqlite_utils/cli.py#L1289-L1292

I can fix this by checking to see if csv_table is already in use and adding a suffix.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 sqlite-utils memory can't deal with multiple files with the same name 990844088  
925301981 https://github.com/simonw/sqlite-utils/issues/325#issuecomment-925301981 https://api.github.com/repos/simonw/sqlite-utils/issues/325 IC_kwDOCGYnMM43Jvzd simonw 9599 2021-09-22T20:23:25Z 2021-09-22T20:23:25Z OWNER

Oddly I can't replicate this on macOS:

(sqlite-utils) sqlite-utils % ls foo/*.csv
foo/bug.csv
(sqlite-utils) sqlite-utils % ls bar/*.csv
bar/bug.csv
(sqlite-utils) sqlite-utils % sqlite-utils memory foo/bug.csv bar/bug.csv --schema
CREATE TABLE "bug" (
   [col1] TEXT,
   [col2] TEXT
);
CREATE VIEW t1 AS select * from [bug];
CREATE VIEW t AS select * from [bug];

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 sqlite-utils memory can't deal with multiple files with the same name 990844088  
925300720 https://github.com/simonw/sqlite-utils/issues/328#issuecomment-925300720 https://api.github.com/repos/simonw/sqlite-utils/issues/328 IC_kwDOCGYnMM43Jvfw gravis 12752 2021-09-22T20:21:33Z 2021-09-22T20:21:33Z NONE

Wow, that was fast! Thank you!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Invalid JSON output when no rows 1004613267  
925300637 https://github.com/simonw/sqlite-utils/issues/325#issuecomment-925300637 https://api.github.com/repos/simonw/sqlite-utils/issues/325 IC_kwDOCGYnMM43Jved simonw 9599 2021-09-22T20:21:26Z 2021-09-22T20:21:26Z OWNER

The t1 and t2 aliases were meant to handle this case, but the are no good if the tool throws an error.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 sqlite-utils memory can't deal with multiple files with the same name 990844088  
925296085 https://github.com/simonw/sqlite-utils/issues/328#issuecomment-925296085 https://api.github.com/repos/simonw/sqlite-utils/issues/328 IC_kwDOCGYnMM43JuXV simonw 9599 2021-09-22T20:14:53Z 2021-09-22T20:14:53Z OWNER 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Invalid JSON output when no rows 1004613267  
925292384 https://github.com/simonw/sqlite-utils/issues/328#issuecomment-925292384 https://api.github.com/repos/simonw/sqlite-utils/issues/328 IC_kwDOCGYnMM43Jtdg simonw 9599 2021-09-22T20:09:53Z 2021-09-22T20:09:53Z OWNER

Good catch, thanks.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Invalid JSON output when no rows 1004613267  
924437942 https://github.com/simonw/datasette/issues/111#issuecomment-924437942 https://api.github.com/repos/simonw/datasette/issues/111 IC_kwDOBm6k_c43Gc22 simonw 9599 2021-09-21T22:32:59Z 2021-09-21T22:47:07Z OWNER

Side-note: Django 4.0 will switch from using pytz to using the standard library zoneinfo module introduced in Python 3.9, which has a backport that works as far back as 3.6: https://github.com/pganssle/zoneinfo (https://pypi.org/project/backports.zoneinfo/)

If I need to handle timezones I'll use that, but I think I can get away without it?

Django does this: https://github.com/django/django/blob/b0ed619303d2fb723330ca9efa3acf23d49f1d19/setup.cfg#L39-L43

install_requires =
    asgiref >= 3.3.2
    backports.zoneinfo; python_version<"3.9"
    sqlparse >= 0.2.2
    tzdata; sys_platform == 'win32'

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add “updated” to metadata 274615452  
924443089 https://github.com/simonw/datasette/issues/111#issuecomment-924443089 https://api.github.com/repos/simonw/datasette/issues/111 IC_kwDOBm6k_c43GeHR simonw 9599 2021-09-21T22:45:14Z 2021-09-21T22:45:26Z OWNER

The audiences I care about here are:

  • Producers of this timestamp - generally that will be users who are using datasette publish to share their data
  • Human consumers of this timestamp - end users who look at a Datasette site and want to know how recent the data is
  • Machine consumers of this timestamp - API integrations that might want to check if a Datasette instance has been updated before downloading new data

For producers I think there are going to be two categories. The first is users who run "publish" and want the site to reflect when they did so (probably using --updated=now when they publish). The second are users who are willing to spend more time thinking about this - for example my various git scraping projects where I want to use a date derived from the git history.

For humans... I'd like to be able to calculate a relative time (3 hours ago) in addition to showing the display time, because that helps avoid confusion over timezones.

For machine consumers, it might be nice to offer the option of a calculated Unix timestamp-since-1970, since those can be easier to work with in some languages than running a full ISO date parser.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add “updated” to metadata 274615452  
924438481 https://github.com/simonw/datasette/issues/111#issuecomment-924438481 https://api.github.com/repos/simonw/datasette/issues/111 IC_kwDOBm6k_c43Gc_R simonw 9599 2021-09-21T22:34:03Z 2021-09-21T22:34:21Z OWNER

The simplest possible version of this: it's always represented as a UTC ISO datetime, like this: 

"updated": "2020-10-31T12:00:00+00:00"

Later versions of Datasette could extend this to handle other timezones or support just the date (though that's a backwards incompatible change so probably better to decide on the date thing right now).

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add “updated” to metadata 274615452  
924435971 https://github.com/simonw/datasette/issues/111#issuecomment-924435971 https://api.github.com/repos/simonw/datasette/issues/111 IC_kwDOBm6k_c43GcYD simonw 9599 2021-09-21T22:29:15Z 2021-09-21T22:29:49Z OWNER

So this is a metadata key called updated which can be applied at the table, database or instance level. It is represented as a .isoformat() timestamp.

Question: should I support just the date - yyyy-mm-dd - in addition to the datetime?

I think so. I can easily imagine situations where the exact time of day that a change was made hasn't been recorded, but the overall date is known.

But in that case, should the updated key sometimes be yyyy-mm-dd and sometimes be the full isoformat datetime? Or should there be an updated_date key that's used for just the date?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add “updated” to metadata 274615452  
924432643 https://github.com/simonw/datasette/issues/111#issuecomment-924432643 https://api.github.com/repos/simonw/datasette/issues/111 IC_kwDOBm6k_c43GbkD simonw 9599 2021-09-21T22:23:23Z 2021-09-21T22:23:23Z OWNER

I'm going to use https://github.com/dateutil/dateutil for this - it's been maintained constantly (by an evolving team of contributors) since 2003 and is a very trustworthy dependency.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add “updated” to metadata 274615452  
924209583 https://github.com/dogsheep/twitter-to-sqlite/pull/59#issuecomment-924209583 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/59 IC_kwDODEm0Qs43FlGv simonw 9599 2021-09-21T17:37:34Z 2021-09-21T17:37:34Z MEMBER

Thanks for this!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Fix for since_id bug, closes #58 984942782  
923979964 https://github.com/simonw/datasette/pull/1476#issuecomment-923979964 https://api.github.com/repos/simonw/datasette/issues/1476 IC_kwDOBm6k_c43EtC8 codecov[bot] 22429695 2021-09-21T13:18:22Z 2021-09-21T13:18:22Z NONE

Codecov Report

Merging #1476 (2eb01ff) into main (b28b6cd) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1476   +/-   ##
=======================================
  Coverage   91.83%   91.83%           
=======================================
  Files          34       34           
  Lines        4421     4421           
=======================================
  Hits         4060     4060           
  Misses        361      361

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b28b6cd...2eb01ff. Read the comment docs.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Update pytest-xdist requirement from <2.4,>=2.2.1 to >=2.2.1,<2.5 1002459220  
923106887 https://github.com/simonw/datasette/issues/111#issuecomment-923106887 https://api.github.com/repos/simonw/datasette/issues/111 IC_kwDOBm6k_c43BX5H simonw 9599 2021-09-20T16:58:39Z 2021-09-20T16:58:39Z OWNER

Still a good idea today too! Would be great for https://cdc-vaccination-history.datasette.io/ for example.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add “updated” to metadata 274615452  
922394999 https://github.com/simonw/datasette/issues/1473#issuecomment-922394999 https://api.github.com/repos/simonw/datasette/issues/1473 IC_kwDOBm6k_c42-qF3 mroswell 192568 2021-09-19T00:44:39Z 2021-09-19T00:45:32Z CONTRIBUTOR

I replaced:

<a class="text-3xl font-bold leading-none" href="https://www.saferortoxic.org">
<img 
src="https://www.saferortoxic.org/images/logo_hu26e4dce8d5931af1ea33526b28fc8383_9734_c52a4f1635ef88bda858373270551ed2.webp" 
class="custom-logo" 
alt="Logo: Safer or Toxic?" 
width="300px">
</a>

with:

<img src="https://www.saferortoxic.org/images/logo_hu26e4dce8d5931af1ea33526b28fc8383_9734_c52a4f1635ef88bda858373270551ed2.webp" 
alt="Logo: Safer or Toxic?" 
style="width:300px; cursor:pointer;" 
onclick="window.location.href='https://saferortoxic.org';return false;">

I'd still love to know what caused this (and how to troubleshoot to figure it out), so I'll leave it open for a bit, but I do have a functional logo linking to the Hugo home page, at least locally. I'll likely push tomorrow.

(Before trying this, I tried to apply a background image to the a tag. That didn't work.)

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 base logo link visits `undefined` rather than href url 999902754  
922363640 https://github.com/simonw/datasette/issues/1473#issuecomment-922363640 https://api.github.com/repos/simonw/datasette/issues/1473 IC_kwDOBm6k_c42-ib4 mroswell 192568 2021-09-18T19:45:47Z 2021-09-18T19:45:47Z CONTRIBUTOR

An update, if I remove the img tag and replace it with the text, "Safer or Toxic?" it links to the right place.

Also, if I keep things exactly as they are, and it improperly, but consistently goes to the undefined page, on THAT 404 page, a click on the image properly clicks through to the www.SaferOrToxic.org page.

Weird stuff.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 base logo link visits `undefined` rather than href url 999902754  
922075480 https://github.com/simonw/datasette/issues/236#issuecomment-922075480 https://api.github.com/repos/simonw/datasette/issues/236 IC_kwDOBm6k_c429cFY simonw 9599 2021-09-17T20:54:13Z 2021-09-17T20:54:13Z OWNER

That's so useful @sethvincent! Really interesting reading your code there, especially clever how you're using the base_url config.

I'd be very interested to see what your demo looks like without using serverless - completely agree that the less additional dependencies there are for this the better.

I'm also very interested in figuring out a way to run Datasette in Lambda but with the SQLite database on an EFS volume. Do you have a feel for how hard that would be?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 datasette publish lambda plugin 317001500  
920543967 https://github.com/simonw/datasette/issues/236#issuecomment-920543967 https://api.github.com/repos/simonw/datasette/issues/236 IC_kwDOBm6k_c423mLf sethvincent 164214 2021-09-16T03:19:08Z 2021-09-16T03:19:08Z NONE

:wave: I just put together a small example using the lambda container image support: https://github.com/sethvincent/datasette-aws-lambda-example

It uses mangum and AWS's python runtime interface client to handle the lambda event stuff.

I'd be happy to help with a publish plugin for AWS lambda as I plan to use this for upcoming projects.

The example uses the serverless cli for deployment but there might be a more suitable deployment approach for the plugin. It would be cool if users didn't have to install anything additional other than the aws cli and its associated config/credentials setup.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 datasette publish lambda plugin 317001500  
919141156 https://github.com/simonw/datasette/pull/1471#issuecomment-919141156 https://api.github.com/repos/simonw/datasette/issues/1471 IC_kwDOBm6k_c42yPsk codecov[bot] 22429695 2021-09-14T13:16:29Z 2021-09-14T13:16:29Z NONE

Codecov Report

Merging #1471 (01b05ed) into main (b28b6cd) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1471   +/-   ##
=======================================
  Coverage   91.83%   91.83%           
=======================================
  Files          34       34           
  Lines        4421     4421           
=======================================
  Hits         4060     4060           
  Misses        361      361

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b28b6cd...01b05ed. Read the comment docs.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Bump black from 21.7b0 to 21.9b0 996002181  
919135732 https://github.com/simonw/datasette/pull/1453#issuecomment-919135732 https://api.github.com/repos/simonw/datasette/issues/1453 IC_kwDOBm6k_c42yOX0 dependabot[bot] 49699333 2021-09-14T13:10:38Z 2021-09-14T13:10:38Z CONTRIBUTOR

Superseded by #1471.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Bump black from 21.7b0 to 21.8b0 982780906  
918621705 https://github.com/simonw/datasette/issues/1464#issuecomment-918621705 https://api.github.com/repos/simonw/datasette/issues/1464 IC_kwDOBm6k_c42wQ4J bobwhitelock 7476523 2021-09-13T22:17:17Z 2021-09-13T22:17:17Z CONTRIBUTOR

haven't had time to get back to this, but idle thought that I'm recording for later investigation: how does the continuous integration handle this installation issue? Is it documented there?

Not certain, but I think tests in CI run on Ubuntu and don't appear to install any additional Sqlite-related dependencies, and so my guess is the version of Sqlite installed by default on Ubuntu has the SQLITE_ENABLE_FTS3_PARENTHESIS option enabled and so doesn't run into this issue.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 clean checkout & clean environment has test failures 991191951  
869105782 https://github.com/simonw/datasette/pull/1385#issuecomment-869105782 https://api.github.com/repos/simonw/datasette/issues/1385 MDEyOklzc3VlQ29tbWVudDg2OTEwNTc4Mg== codecov[bot] 22429695 2021-06-27T05:48:55Z 2021-09-13T17:29:30Z NONE

Codecov Report

Merging #1385 (db78094) into main (ea627ba) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head db78094 differs from pull request most recent head 8d78c8c. Consider uploading reports for the commit 8d78c8c to get more accurate results

@@           Coverage Diff           @@
##             main    #1385   +/-   ##
=======================================
  Coverage   91.70%   91.70%           
=======================================
  Files          34       34           
  Lines        4364     4364           
=======================================
  Hits         4002     4002           
  Misses        362      362

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ea627ba...8d78c8c. Read the comment docs.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Fix + improve get_metadata plugin hook docs 930855052  
917840012 https://github.com/simonw/datasette/issues/1466#issuecomment-917840012 https://api.github.com/repos/simonw/datasette/issues/1466 IC_kwDOBm6k_c42tSCM simonw 9599 2021-09-13T04:54:59Z 2021-09-13T04:54:59Z OWNER

Especially relevant now that 0.2.0 is out which is a much higher quality release. https://github.com/simonw/datasette-app/releases/tag/0.2.0

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add Datasette Desktop to installation documentation 991467558  
917839801 https://github.com/simonw/datasette/issues/1468#issuecomment-917839801 https://api.github.com/repos/simonw/datasette/issues/1468 IC_kwDOBm6k_c42tR-5 simonw 9599 2021-09-13T04:54:17Z 2021-09-13T04:54:17Z OWNER

Here's a already open issue for this: #972

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Faceting for custom SQL queries 994390593  
917839507 https://github.com/simonw/datasette/issues/1468#issuecomment-917839507 https://api.github.com/repos/simonw/datasette/issues/1468 IC_kwDOBm6k_c42tR6T simonw 9599 2021-09-13T04:53:22Z 2021-09-13T04:53:22Z OWNER

At the moment this isn't possible - though there's a workaround which is to define a SQL view for the query, at which point facets will be displayed again.

I did a lot of the work required to support this when I refactored how facets worked a while back - but to finally implement this I need to refactor the table view and the arbitrary query view to share much more logic than they do at the moment.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Faceting for custom SQL queries 994390593  
917839062 https://github.com/simonw/datasette/issues/1469#issuecomment-917839062 https://api.github.com/repos/simonw/datasette/issues/1469 IC_kwDOBm6k_c42tRzW simonw 9599 2021-09-13T04:52:01Z 2021-09-13T04:52:01Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Column cog shows "facet by this" when already default faceted 994450961  
917642487 https://github.com/simonw/datasette/issues/1464#issuecomment-917642487 https://api.github.com/repos/simonw/datasette/issues/1464 IC_kwDOBm6k_c42shz3 ctb 51016 2021-09-12T14:03:09Z 2021-09-12T14:03:09Z NONE

haven't had time to get back to this, but idle thought that I'm recording for later investigation: how does the continuous integration handle this installation issue? Is it documented there?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 clean checkout & clean environment has test failures 991191951  
916119657 https://github.com/simonw/sqlite-utils/pull/326#issuecomment-916119657 https://api.github.com/repos/simonw/sqlite-utils/issues/326 IC_kwDOCGYnMM42muBp meatcar 191622 2021-09-09T13:54:10Z 2021-09-09T13:54:10Z NONE

dupe of #293?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Test against 3.10-dev 991237645  
915343886 https://github.com/simonw/datasette/issues/1464#issuecomment-915343886 https://api.github.com/repos/simonw/datasette/issues/1464 IC_kwDOBm6k_c42jwoO bobwhitelock 7476523 2021-09-08T15:32:06Z 2021-09-08T15:32:06Z CONTRIBUTOR

Thanks, that does look similar!

Unfortunately, pysqlite3-binary isn't available for Mac OS X, so I can't quickly check that that fixes it; will do so later.

Ah that makes sense, I guess that's why this isn't just always installed already. I wonder if a possible solution to this issue could be doing feature detection on whether this feature is supported by the current Sqlite version, and if not these tests could be disabled locally? But possibly there's a better way to handle this, will see what @simonw thinks

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 clean checkout & clean environment has test failures 991191951  
915321467 https://github.com/simonw/sqlite-utils/pull/326#issuecomment-915321467 https://api.github.com/repos/simonw/sqlite-utils/issues/326 IC_kwDOCGYnMM42jrJ7 codecov[bot] 22429695 2021-09-08T15:05:53Z 2021-09-08T15:05:53Z NONE

Codecov Report

Merging #326 (078a087) into main (77c240d) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #326   +/-   ##
=======================================
  Coverage   96.58%   96.58%           
=======================================
  Files           5        5           
  Lines        2223     2223           
=======================================
  Hits         2147     2147           
  Misses         76       76

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 77c240d...078a087. Read the comment docs.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Test against 3.10-dev 991237645  
915302885 https://github.com/simonw/datasette/issues/1464#issuecomment-915302885 https://api.github.com/repos/simonw/datasette/issues/1464 IC_kwDOBm6k_c42jmnl ctb 51016 2021-09-08T14:44:50Z 2021-09-08T14:44:50Z NONE

thanks for the response! full errors attached; excerpt:

...


    def test_searchmode(table_metadata, querystring, expected_rows):
        with make_app_client(
            metadata={"databases": {"fixtures": {"tables": {"searchable": table_metadata}}}}
        ) as client:
            response = client.get("/fixtures/searchable.json?" + querystring)
>           assert expected_rows == response.json["rows"]
E           AssertionError: assert [[1, 'barry c...sel', 'puma']] == []
E             Left contains 2 more items, first extra item: [1, 'barry cat', 'terry dog', 'panther']
E             Use -v to get the full diff

/Users/t/dev/datasette/tests/test_api.py:1115: AssertionError

errors.txt

A quick scan of #1223 suggests you're right. Unfortunately, pysqlite3-binary isn't available for Mac OS X, so I can't quickly check that that fixes it; will do so later.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 clean checkout & clean environment has test failures 991191951  
915299013 https://github.com/simonw/datasette/issues/1464#issuecomment-915299013 https://api.github.com/repos/simonw/datasette/issues/1464 IC_kwDOBm6k_c42jlrF bobwhitelock 7476523 2021-09-08T14:40:28Z 2021-09-08T14:40:28Z CONTRIBUTOR

What are the full errors you're getting?

This may be the same issue as described in https://github.com/simonw/datasette/pull/1223 - essentially the test suite (and corresponding Datasette features I assume) are by default implicitly dependent on your Sqlite installation having been compiled with the SQLITE_ENABLE_FTS3_PARENTHESIS option. If this is the same issue then I think this can be fixed either by recompiling with that option or (probably more easily) by running pip install pysqlite3-binary, which will be used in preference to your system Sqlite installation and has this option enabled.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 clean checkout & clean environment has test failures 991191951  
915279711 https://github.com/simonw/datasette/issues/1464#issuecomment-915279711 https://api.github.com/repos/simonw/datasette/issues/1464 IC_kwDOBm6k_c42jg9f ctb 51016 2021-09-08T14:16:49Z 2021-09-08T14:16:49Z NONE

on commit d57ab156b35ec642

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 clean checkout & clean environment has test failures 991191951  
915229323 https://github.com/simonw/datasette/pull/1463#issuecomment-915229323 https://api.github.com/repos/simonw/datasette/issues/1463 IC_kwDOBm6k_c42jUqL codecov[bot] 22429695 2021-09-08T13:15:26Z 2021-09-08T13:15:26Z NONE

Codecov Report

Merging #1463 (2dd94c5) into main (d57ab15) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1463   +/-   ##
=======================================
  Coverage   91.83%   91.83%           
=======================================
  Files          34       34           
  Lines        4421     4421           
=======================================
  Hits         4060     4060           
  Misses        361      361

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d57ab15...2dd94c5. Read the comment docs.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Update beautifulsoup4 requirement from <4.10.0,>=4.8.1 to >=4.8.1,<4.11.0 991121619  
914644260 https://github.com/simonw/datasette/issues/1462#issuecomment-914644260 https://api.github.com/repos/simonw/datasette/issues/1462 IC_kwDOBm6k_c42hF0k simonw 9599 2021-09-07T21:34:32Z 2021-09-07T21:34:32Z OWNER

I think this is a setting. There are two relevant settings at the moment:

    "template_debug": false,
    "trace_debug": false,

For consistence then this should be called something_debug - but do I want a single setting that exposes the _internal database and adds those debug options to the menu, or do I want those as two separate settings?

  • internal_debug to enable access to that _internal database
  • menu_debug for those menu options?
 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Separate out "debug" options from "root" options 990367646  
914441037 https://github.com/simonw/datasette/issues/1461#issuecomment-914441037 https://api.github.com/repos/simonw/datasette/issues/1461 IC_kwDOBm6k_c42gUNN simonw 9599 2021-09-07T16:13:59Z 2021-09-07T16:13:59Z OWNER

I don't think I'll adopt it for this project. For example, here:

     response = Response.redirect("/")
-    response.set_cookie("ds_actor", datasette.sign({
-        "a": {
-            "id": "cleopaws"
-        }
-    }, "actor"))
+    response.set_cookie("ds_actor", datasette.sign({"a": {"id": "cleopaws"}}, "actor"))

I chose to use the multi-line version to help emphasize the structure - the single-line replacement loses that. I think I'll continue to make my own editorial choices about how the code examples are laid out.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Try blacken-docs 989986586  
914440282 https://github.com/simonw/datasette/issues/1461#issuecomment-914440282 https://api.github.com/repos/simonw/datasette/issues/1461 IC_kwDOBm6k_c42gUBa simonw 9599 2021-09-07T16:12:57Z 2021-09-07T16:12:57Z OWNER

Here's the diff it produced from that first run:
`diff diff --git a/docs/authentication.rst b/docs/authentication.rst index 0d98cf8..8008023 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -381,11 +381,7 @@ Authentication plugins can set signedds_actor`` cookies themselves like so:
.. code-block:: python

 response = Response.redirect("/")
  • response.set_cookie("ds_actor", datasette.sign({
  • "a": {
  • "id": "cleopaws"
  • }
  • }, "actor"))
  • response.set_cookie("ds_actor", datasette.sign({"a": {"id": "cleopaws"}}, "actor"))

Note that you need to pass "actor" as the namespace to :ref:datasette_sign.

@@ -412,12 +408,16 @@ To include an expiry, add a "e" key to the cookie value containing a `base62
expires_at = int(time.time()) + (24 * 60 * 60)

 response = Response.redirect("/")
  • response.set_cookie("ds_actor", datasette.sign({
  • "a": {
  • "id": "cleopaws"
  • },
  • "e": baseconv.base62.encode(expires_at),
  • }, "actor"))
  • response.set_cookie(
  • "ds_actor",
  • datasette.sign(
  • {
  • "a": {"id": "cleopaws"},
  • "e": baseconv.base62.encode(expires_at),
  • },
  • "actor",
  • ),
  • )

The resulting cookie will encode data that looks something like this:

diff --git a/docs/spatialite.rst b/docs/spatialite.rst
index d1b300b..556bad8 100644
--- a/docs/spatialite.rst
+++ b/docs/spatialite.rst
@@ -58,19 +58,22 @@ Here's a recipe for taking a table with existing latitude and longitude columns,
.. code-block:: python

 import sqlite3
  • conn = sqlite3.connect('museums.db')
    +
  • conn = sqlite3.connect("museums.db")
    # Lead the spatialite extension:
    conn.enable_load_extension(True)
  • conn.load_extension('/usr/local/lib/mod_spatialite.dylib')
  • conn.load_extension("/usr/local/lib/mod_spatialite.dylib")
    # Initialize spatial metadata for this database:
  • conn.execute('select InitSpatialMetadata(1)')
  • conn.execute("select InitSpatialMetadata(1)")
    # Add a geometry column called point_geom to our museums table:
    conn.execute("SELECT AddGeometryColumn('museums', 'point_geom', 4326, 'POINT', 2);")
    # Now update that geometry column with the lat/lon points
  • conn.execute('''
  • conn.execute(
  • """
    UPDATE museums SET
    point_geom = GeomFromText('POINT('||"longitude"||' '||"latitude"||')',4326);
  • ''')
  • """

  • )
    # Now add a spatial index to that column
    conn.execute('select CreateSpatialIndex("museums", "point_geom");')
    # If you don't commit your changes will not be persisted:
    @@ -186,13 +189,14 @@ Here's Python code to create a SQLite database, enable SpatiaLite, create a plac
    .. code-block:: python

    import sqlite3
    - conn = sqlite3.connect('places.db')
    +
    + conn = sqlite3.connect("places.db")
    # Enable SpatialLite extension
    conn.enable_load_extension(True)
    - conn.load_extension('/usr/local/lib/mod_spatialite.dylib')
    + conn.load_extension("/usr/local/lib/mod_spatialite.dylib")
    # Create the masic countries table
    - conn.execute('select InitSpatialMetadata(1)')
    - conn.execute('create table places (id integer primary key, name text);')
    + conn.execute("select InitSpatialMetadata(1)")
    + conn.execute("create table places (id integer primary key, name text);")
    # Add a MULTIPOLYGON Geometry column
    conn.execute("SELECT AddGeometryColumn('places', 'geom', 4326, 'MULTIPOLYGON', 2);")
    # Add a spatial index against the new column
    @@ -201,13 +205,17 @@ Here's Python code to create a SQLite database, enable SpatiaLite, create a plac
    from shapely.geometry.multipolygon import MultiPolygon
    from shapely.geometry import shape
    import requests
    - geojson = requests.get('https://data.whosonfirst.org/404/227/475/404227475.geojson').json()
    +
    + geojson = requests.get(
    + "https://data.whosonfirst.org/404/227/475/404227475.geojson"
    + ).json()
    # Convert to "Well Known Text" format
    - wkt = shape(geojson['geometry']).wkt
    + wkt = shape(geojson["geometry"]).wkt
    # Insert and commit the record
    - conn.execute("INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))", (
    - "Wales", wkt
    - ))
    + conn.execute(
    + "INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))",
    + ("Wales", wkt),
    + )
    conn.commit()

Querying polygons using within()
diff --git a/docs/writing_plugins.rst b/docs/writing_plugins.rst
index bd60a4b..5af01f6 100644
--- a/docs/writing_plugins.rst
+++ b/docs/writing_plugins.rst
@@ -18,9 +18,10 @@ The quickest way to start writing a plugin is to create a my_plugin.py file

 from datasette import hookimpl

+
@hookimpl
def prepare_connection(conn):
- conn.create_function('hello_world', 0, lambda: 'Hello world!')
+ conn.create_function("hello_world", 0, lambda: "Hello world!")

If you save this in plugins/my_plugin.py you can then start Datasette like this::

@@ -60,22 +61,18 @@ The example consists of two files: a setup.py file that defines the plugin:

 from setuptools import setup
  • VERSION = '0.1'

  • VERSION = "0.1"

    setup(
    - name='datasette-plugin-demos',
    - description='Examples of plugins for Datasette',
    - author='Simon Willison',
    - url='https://github.com/simonw/datasette-plugin-demos',
    - license='Apache License, Version 2.0',
    + name="datasette-plugin-demos",
    + description="Examples of plugins for Datasette",
    + author="Simon Willison",
    + url="https://github.com/simonw/datasette-plugin-demos",
    + license="Apache License, Version 2.0",
    version=VERSION,
    - py_modules=['datasette_plugin_demos'],
    - entry_points={
    - 'datasette': [
    - 'plugin_demos = datasette_plugin_demos'
    - ]
    - },
    - install_requires=['datasette']
    + py_modules=["datasette_plugin_demos"],
    + entry_points={"datasette": ["plugin_demos = datasette_plugin_demos"]},
    + install_requires=["datasette"],
    )

And a Python module file, datasette_plugin_demos.py, that implements the plugin:
@@ -88,12 +85,12 @@ And a Python module file, datasette_plugin_demos.py, that implements the plu

 @hookimpl
 def prepare_jinja2_environment(env):
  • env.filters['uppercase'] = lambda u: u.upper()

  • env.filters["uppercase"] = lambda u: u.upper()

    @hookimpl
    def prepare_connection(conn):
    - conn.create_function('random_integer', 2, random.randint)
    + conn.create_function("random_integer", 2, random.randint)

Having built a plugin in this way you can turn it into an installable package using the following command::
@@ -123,11 +120,13 @@ To bundle the static assets for a plugin in the package that you publish to PyPI

.. code-block:: python

  • package_data={
  • 'datasette_plugin_name': [
  • 'static/plugin.js',
  • ],
  • },
  • package_data = (
  • {
  • "datasette_plugin_name": [
  • "static/plugin.js",
  • ],
  • },
  • )

Where datasette_plugin_name is the name of the plugin package (note that it uses underscores, not hyphens) and static/plugin.js is the path within that package to the static file.

@@ -152,11 +151,13 @@ Templates should be bundled for distribution using the same package_data mec

.. code-block:: python

  • package_data={
  • 'datasette_plugin_name': [
  • 'templates/my_template.html',
  • ],
  • },
  • package_data = (
  • {
  • "datasette_plugin_name": [
  • "templates/my_template.html",
  • ],
  • },
  • )

You can also use wildcards here such as templates/*.html. See datasette-edit-schema <https://github.com/simonw/datasette-edit-schema>__ for an example of this pattern.
```

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Try blacken-docs 989986586  
914439356 https://github.com/simonw/datasette/issues/1461#issuecomment-914439356 https://api.github.com/repos/simonw/datasette/issues/1461 IC_kwDOBm6k_c42gTy8 simonw 9599 2021-09-07T16:11:37Z 2021-09-07T16:11:37Z OWNER 
(datasette) datasette % blacken-docs docs/*.rst
docs/authentication.rst: Rewriting...
docs/internals.rst:169: code block parse error Cannot parse: 14:0: <line number missing in source>
docs/plugin_hooks.rst:251: code block parse error Cannot parse: 6:4:     ]
docs/plugin_hooks.rst:312: code block parse error Cannot parse: 38:0: <line number missing in source>
docs/spatialite.rst: Rewriting...
docs/testing_plugins.rst:135: code block parse error Cannot parse: 5:0: <line number missing in source>
docs/writing_plugins.rst: Rewriting...

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Try blacken-docs 989986586  
913218494 https://github.com/simonw/datasette/issues/1459#issuecomment-913218494 https://api.github.com/repos/simonw/datasette/issues/1459 IC_kwDOBm6k_c42bpu- simonw 9599 2021-09-05T19:58:51Z 2021-09-05T19:59:15Z OWNER

This idea makes sense to me.

There's actually an existing option that takes a path, called --get - it returns the HTML or JSON for that oath directly to the console, eg datasette my.db --get /mydb/mytable.json

So... one option would be to allow combining that with -o to open that URL in the browser:

datasette my.db -o --get /mydb

So some options here are:

  • datasette my.db --open-url /mydb
  • datasette my.db --open-path /mydb
  • datasette my.db --open --get /mydb

I quite like that last combination option, mainly to avoid adding even more command options.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 suggestion: allow `datasette --open` to take a relative URL 988556488  
913001416 https://github.com/simonw/datasette/pull/1455#issuecomment-913001416 https://api.github.com/repos/simonw/datasette/issues/1455 IC_kwDOBm6k_c42a0vI simonw 9599 2021-09-04T16:32:21Z 2021-09-04T16:32:21Z OWNER

I'll add researchers too.

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add scientists to target groups 988325628  
913001298 https://github.com/simonw/datasette/pull/1455#issuecomment-913001298 https://api.github.com/repos/simonw/datasette/issues/1455 IC_kwDOBm6k_c42a0tS simonw 9599 2021-09-04T16:31:32Z 2021-09-04T16:31:32Z OWNER

Great idea!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add scientists to target groups 988325628  
913001282 https://github.com/simonw/datasette/pull/1455#issuecomment-913001282 https://api.github.com/repos/simonw/datasette/issues/1455 IC_kwDOBm6k_c42a0tC ctb 51016 2021-09-04T16:31:24Z 2021-09-04T16:31:24Z NONE

I love it! maybe 'researchers' instead? Or 'scientists and researchers'?

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add scientists to target groups 988325628  
911772943 https://github.com/dogsheep/evernote-to-sqlite/issues/14#issuecomment-911772943 https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/14 IC_kwDOEhK-wc42WI0P step21 46968 2021-09-02T14:53:11Z 2021-09-02T14:53:11Z NONE

Additionally, assuming the line numbers match up with the provided enenx file, the mentioned line plus one before and after is as follows:

<![CDATA[>]]>
</span></div>
<div style="padding: 0px; font-family: Arial, sans-serif; font-size: 12px; line-height: 16px; white-space: pre-wrap;"><br style=" padding: 0px;"/>

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 xml.etree.ElementTree.Parse Error - mismatched tag 986829194  
910121331 https://github.com/dogsheep/twitter-to-sqlite/issues/58#issuecomment-910121331 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/58 IC_kwDODEm0Qs42P1lz rubenv 42904 2021-09-01T09:49:33Z 2021-09-01T09:49:33Z CONTRIBUTOR

Found the cause, it's the other commands. PR #59 submitted.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Error: Use either --since or --since_id, not both - still broken 984939366  
908832938 https://github.com/simonw/datasette/issues/1446#issuecomment-908832938 https://api.github.com/repos/simonw/datasette/issues/1446 IC_kwDOBm6k_c42K7Cq simonw 9599 2021-08-31T01:54:59Z 2021-08-31T01:54:59Z OWNER

I used the sticky footer mechanism in datasette.app: https://github.com/simonw/datasette.app/issues/3

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Modify base.html template to support optional sticky footer 978357984  
908337614 https://github.com/simonw/datasette/pull/1453#issuecomment-908337614 https://api.github.com/repos/simonw/datasette/issues/1453 IC_kwDOBm6k_c42JCHO codecov[bot] 22429695 2021-08-30T13:20:26Z 2021-08-30T13:20:26Z NONE

Codecov Report

Merging #1453 (4f492a7) into main (67cbf0a) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1453   +/-   ##
=======================================
  Coverage   91.83%   91.83%           
=======================================
  Files          34       34           
  Lines        4421     4421           
=======================================
  Hits         4060     4060           
  Misses        361      361

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 67cbf0a...4f492a7. Read the comment docs.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Bump black from 21.7b0 to 21.8b0 982780906  
907547624 https://github.com/simonw/datasette/issues/1449#issuecomment-907547624 https://api.github.com/repos/simonw/datasette/issues/1449 IC_kwDOBm6k_c42GBPo simonw 9599 2021-08-28T01:44:57Z 2021-08-28T01:58:35Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `register_commands()` plugin hook to register extra CLI commands 981676832  
907547736 https://github.com/simonw/datasette/issues/1449#issuecomment-907547736 https://api.github.com/repos/simonw/datasette/issues/1449 IC_kwDOBm6k_c42GBRY simonw 9599 2021-08-28T01:45:36Z 2021-08-28T01:45:36Z OWNER

I need to push this out as an alpha so I can release a demo plugin that uses it.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `register_commands()` plugin hook to register extra CLI commands 981676832  
907543982 https://github.com/simonw/datasette/issues/1449#issuecomment-907543982 https://api.github.com/repos/simonw/datasette/issues/1449 IC_kwDOBm6k_c42GAWu simonw 9599 2021-08-28T01:14:27Z 2021-08-28T01:14:27Z OWNER

Writing the test for this is proving difficult, because the cli module has already been imported when I attempt to register a new plugin - so it doesn't pick up on the additional command registrations.

Trying to work around that with importlib.reload(cli).

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `register_commands()` plugin hook to register extra CLI commands 981676832  
907542214 https://github.com/simonw/datasette/issues/1449#issuecomment-907542214 https://api.github.com/repos/simonw/datasette/issues/1449 IC_kwDOBm6k_c42F_7G simonw 9599 2021-08-28T01:02:38Z 2021-08-28T01:02:38Z OWNER

Partial prototype of datasette-verify:

from datasette import hookimpl
import click


@hookimpl
def register_commands(cli):
    from datasette.cli import sqlite_extensions
    @cli.command()
    @click.argument("files", type=click.Path(exists=True), nargs=-1)
    @sqlite_extensions
    def verify(files, sqlite_extensions):
        "Verify that files can be opened by Datasette"
        for file in files:
            print(file)

I had to move the from datasette.cli import sqlite_extensions inside the hook function to avoid a circular import.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `register_commands()` plugin hook to register extra CLI commands 981676832  
907540928 https://github.com/simonw/datasette/issues/1449#issuecomment-907540928 https://api.github.com/repos/simonw/datasette/issues/1449 IC_kwDOBm6k_c42F_nA simonw 9599 2021-08-28T00:53:37Z 2021-08-28T00:53:37Z OWNER

I'll probably have to use this mechanism for the tests then: https://til.simonwillison.net/pytest/registering-plugins-in-tests

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `register_commands()` plugin hook to register extra CLI commands 981676832  
907540790 https://github.com/simonw/datasette/issues/1449#issuecomment-907540790 https://api.github.com/repos/simonw/datasette/issues/1449 IC_kwDOBm6k_c42F_k2 simonw 9599 2021-08-28T00:52:32Z 2021-08-28T00:52:32Z OWNER

I don't think I can get this new hook to support the handy --plugins-dir= mechanism for loading plugins from Python files as opposed to registering them with setuptools - that mechanism is itself implemented inside of code called by datasette serve so I don't have a way of taking advantage of it from outside that command.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `register_commands()` plugin hook to register extra CLI commands 981676832  
907540240 https://github.com/simonw/datasette/issues/1450#issuecomment-907540240 https://api.github.com/repos/simonw/datasette/issues/1450 IC_kwDOBm6k_c42F_cQ simonw 9599 2021-08-28T00:48:30Z 2021-08-28T00:48:30Z OWNER

I'll go with this:

% datasette --help
Usage: datasette [OPTIONS] COMMAND [ARGS]...

  Datasette is an open source multi-tool for exploring and publishing data

  About Datasette: https://datasette.io/
  Full documentation: https://docs.datasette.io/

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Datasette --help should show something more useful than "Datasette!" 981681138  
907539668 https://github.com/simonw/datasette/issues/1449#issuecomment-907539668 https://api.github.com/repos/simonw/datasette/issues/1449 IC_kwDOBm6k_c42F_TU simonw 9599 2021-08-28T00:44:16Z 2021-08-28T00:44:16Z OWNER

Considering this piece of code: https://github.com/simonw/datasette/blob/a1a33bb5822214be1cebd98cd858b2058d91a4aa/datasette/cli.py#L122-L142

I think the hook itself gets called with a single argument, cli, which it can then use in the standard Click way to register extra stuff.

I can't pass it datasette (like I do with register_routes()) because the Datasette object itself is instantiated by the serve command, which will not have been called.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `register_commands()` plugin hook to register extra CLI commands 981676832  
907539251 https://github.com/simonw/datasette/issues/1449#issuecomment-907539251 https://api.github.com/repos/simonw/datasette/issues/1449 IC_kwDOBm6k_c42F_Mz simonw 9599 2021-08-28T00:41:37Z 2021-08-28T00:41:50Z OWNER

The first example plugin I'm going to build for this will be datasette verify file.db file2.db - it will take one or more paths to SQLite files and verify if they can be opened by Datasette or not.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `register_commands()` plugin hook to register extra CLI commands 981676832  
907539065 https://github.com/simonw/datasette/issues/1449#issuecomment-907539065 https://api.github.com/repos/simonw/datasette/issues/1449 IC_kwDOBm6k_c42F_J5 simonw 9599 2021-08-28T00:40:24Z 2021-08-28T00:40:24Z OWNER

I'm going to call the new hook register_commands - since it will allow ambitious plugins to register more than one command if they want to. That's also pleasingly similar to register_routes.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `register_commands()` plugin hook to register extra CLI commands 981676832  
907538940 https://github.com/simonw/datasette/issues/1449#issuecomment-907538940 https://api.github.com/repos/simonw/datasette/issues/1449 IC_kwDOBm6k_c42F_H8 simonw 9599 2021-08-28T00:39:28Z 2021-08-28T00:39:28Z OWNER

Here's the answer to that:

~ % datasette --help
Usage: datasette [OPTIONS] COMMAND [ARGS]...

  Datasette!

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  serve*     Serve up specified SQLite database files with a web UI
  inspect
  install    Install Python packages - e.g.
  package    Package specified SQLite files into a new datasette Docker...
  plugins    List currently available plugins
  publish    Publish specified SQLite database files to the internet...
  uninstall  Uninstall Python packages (e.g.

Since it's adding extra things that show up in --help under the "Commands:" heading, I should call them commands.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `register_commands()` plugin hook to register extra CLI commands 981676832  
907537693 https://github.com/simonw/datasette/issues/1449#issuecomment-907537693 https://api.github.com/repos/simonw/datasette/issues/1449 IC_kwDOBm6k_c42F-0d simonw 9599 2021-08-28T00:31:26Z 2021-08-28T00:31:26Z OWNER

Terminology question: is it correct to call these subcommands or should they be commands? publish_subcommand() adds subcommands of the format datasette publish X - but are we instead adding commands with this new one?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `register_commands()` plugin hook to register extra CLI commands 981676832  
907537610 https://github.com/simonw/datasette/issues/1449#issuecomment-907537610 https://api.github.com/repos/simonw/datasette/issues/1449 IC_kwDOBm6k_c42F-zK simonw 9599 2021-08-28T00:30:51Z 2021-08-28T00:30:51Z OWNER

There's also the option for plugins to muck around with existing registered commands - this could get a bit untidy if multiple plugins try to do it, but being able to replace serve with a fresh implementation that adds an additional command-line option before calling back to the original might open up some interesting possibilities.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `register_commands()` plugin hook to register extra CLI commands 981676832  
907537366 https://github.com/simonw/datasette/issues/1449#issuecomment-907537366 https://api.github.com/repos/simonw/datasette/issues/1449 IC_kwDOBm6k_c42F-vW simonw 9599 2021-08-28T00:29:16Z 2021-08-28T00:29:29Z OWNER

The closest plugin hook to this right now is publish_subcommand - which looks like this:

@hookimpl
def publish_subcommand(publish):
    @publish.command()
    @add_common_publish_arguments_and_options
    @click.option(
        "-k",
        "--api_key",
        help="API key for talking to my hosting provider",
    )
    def my_hosting_provider(...):

But there are also several plugin hooks with register_ prefixes, which may be a good naming convention to stick to here: register_output_renderer(datasette), register_routes(datasette), register_facet_classes(), register_magic_parameters(datasette).

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `register_commands()` plugin hook to register extra CLI commands 981676832  
906646452 https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-906646452 https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13 IC_kwDOEhK-wc42ClO0 simonw 9599 2021-08-26T18:34:34Z 2021-08-26T18:35:20Z MEMBER

I tried this ampersand fix: https://regex101.com/r/ojU2H9/1

# https://regex101.com/r/ojU2H9/1
_invalid_ampersand_re = re.compile(r'&(?![a-z0-9]+;)')

def fix_bad_xml(xml):
    # More fixes for things like '&' not as part of an entity
    return _invalid_ampersand_re.sub('&amp;', xml)

Even with that I'm still getting total garbage in the <en-note> content - it's just HTML, not even trying to be XML.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 xml.etree.ElementTree.ParseError: not well-formed (invalid token) 978743426  
906635938 https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-906635938 https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13 IC_kwDOEhK-wc42Ciqi simonw 9599 2021-08-26T18:18:27Z 2021-08-26T18:18:27Z MEMBER

It looks like I was using the round-trip to dump the <?xml version="1.0" encoding="UTF-8" standalone="no"?> and <!DOCTYPE prefixes.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 xml.etree.ElementTree.ParseError: not well-formed (invalid token) 978743426  
906414811 https://github.com/simonw/datasette/pull/1448#issuecomment-906414811 https://api.github.com/repos/simonw/datasette/issues/1448 IC_kwDOBm6k_c42Bsrb codecov[bot] 22429695 2021-08-26T13:29:02Z 2021-08-26T13:29:02Z NONE

Codecov Report

Merging #1448 (a211747) into main (a1a33bb) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1448   +/-   ##
=======================================
  Coverage   91.82%   91.82%           
=======================================
  Files          34       34           
  Lines        4418     4418           
=======================================
  Hits         4057     4057           
  Misses        361      361

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a1a33bb...a211747. Read the comment docs.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Update pluggy requirement from ~=0.13.0 to >=0.13,<1.1 980228553  
906015471 https://github.com/dogsheep/dogsheep-photos/issues/7#issuecomment-906015471 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/7 IC_kwDOD079W842ALLv dkam 18232 2021-08-26T02:01:01Z 2021-08-26T02:01:01Z NONE

Perceptual hashes might be what you're after : http://phash.org

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Integrate image content hashing 602585497  
905904540 https://github.com/simonw/datasette/issues/859#issuecomment-905904540 https://api.github.com/repos/simonw/datasette/issues/859 IC_kwDOBm6k_c41_wGc brandonrobertz 2670795 2021-08-25T21:59:14Z 2021-08-25T21:59:55Z CONTRIBUTOR

I did two tests: one with 1000 5-30mb DBs and a second with 20 multi gig DBs. For the second, I created them like so:
for i in {1..20}; do sqlite-generate db$i.db --tables ${i}00 --rows 100,2000 --columns 5,100 --pks 0 --fks 0; done

This was for deciding whether to use lots of small DBs or to group things into a smaller number of bigger DBs. The second strategy wins.

By simply persisting the _internal DB to disk, I was able to avoid most of the performance issues I was experiencing previously. (To do this, I changed the datasette/internal_db.py:init_internal_db creates to if not exists, and changed the _internal DB instantiation in datasette/app.py:Datasette.__init__ to a path with is_mutable=True.) Super rough, but the pages now load so I can continue testing ideas.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Database page loads too slowly with many large tables (due to table counts) 642572841  
905900807 https://github.com/simonw/datasette/issues/859#issuecomment-905900807 https://api.github.com/repos/simonw/datasette/issues/859 IC_kwDOBm6k_c41_vMH simonw 9599 2021-08-25T21:51:10Z 2021-08-25T21:51:10Z OWNER

10-20 minutes to populate _internal! How many databases and tables is that for?

I may have to rethink the _internal mechanism entirely. One possible alternative would be for the Datasette homepage to just show a list of available databases (maybe only if there are more than X connected) and then load in their metadata only the first time they are accessed.

I need to get my own stress testing rig setup for this.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Database page loads too slowly with many large tables (due to table counts) 642572841  
905899177 https://github.com/simonw/datasette/issues/859#issuecomment-905899177 https://api.github.com/repos/simonw/datasette/issues/859 IC_kwDOBm6k_c41_uyp brandonrobertz 2670795 2021-08-25T21:48:00Z 2021-08-25T21:48:00Z CONTRIBUTOR

Upon first stab, there's two issues here:
- DB/table/row counts (as discussed above). This isn't too bad if the DBs are actually above the MAX limit check.
- Populating the internal DB. On first load of a giant set of DBs, it can take 10-20 mins to populate. By altering datasette and persisting the internal DB to disk, this problem is vastly improved, but I'm sure this will cause problems elsewhere.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Database page loads too slowly with many large tables (due to table counts) 642572841  
905886797 https://github.com/simonw/sqlite-utils/issues/323#issuecomment-905886797 https://api.github.com/repos/simonw/sqlite-utils/issues/323 IC_kwDOCGYnMM41_rxN simonw 9599 2021-08-25T21:25:18Z 2021-08-25T21:25:18Z OWNER

As far as I can tell the Python sqlite3 module doesn't actually have a mechanism for de-registering a custom SQL function.

This means that if I implement a mechanism whereby each call to .convert() registers a new SQL function with a random suffix (convert_value_23424() for example) those functions will stay registered - and if .convert() is called a large number of times the number of obsolete custom function registrations will grow without bounds.

For that reason, I'm going to wontfix this issue.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `table.convert()` method should clean up after itself 979627285  
905206234 https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-905206234 https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13 IC_kwDOEhK-wc419Fna simonw 9599 2021-08-25T05:58:42Z 2021-08-25T05:58:42Z MEMBER

https://github.com/dogsheep/evernote-to-sqlite/blob/36a466f142e5bad52719851c2fbda0c05cd35b99/evernote_to_sqlite/utils.py#L34-L42

Not sure why I was round-tripping the content_xml like that - I will try not doing that.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 xml.etree.ElementTree.ParseError: not well-formed (invalid token) 978743426  
905203570 https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-905203570 https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13 IC_kwDOEhK-wc419E9y simonw 9599 2021-08-25T05:51:22Z 2021-08-25T05:53:27Z MEMBER

The debugger showed me that it broke on a string that looked like this:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE en-note SYSTEM "http://xml.evernote.com/pub/enml2.dtd">
<en-note>
  <h1 title="Q3 2018 Reflection & Development">
    <span title=Q3 2018 Reflection & Development">
      Q3 2018 Reflection & Development
    </span>
  </h1>
  ...

Yeah that is not valid XML!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 xml.etree.ElementTree.ParseError: not well-formed (invalid token) 978743426  
905097468 https://github.com/simonw/datasette/pull/1447#issuecomment-905097468 https://api.github.com/repos/simonw/datasette/issues/1447 IC_kwDOBm6k_c418rD8 simonw 9599 2021-08-25T01:28:53Z 2021-08-25T01:28:53Z OWNER

Good catch, thanks!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Remove underscore from search mode parameter name 978614898  
905043974 https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905043974 https://api.github.com/repos/simonw/sqlite-utils/issues/319 IC_kwDOCGYnMM418eAG simonw 9599 2021-08-24T23:33:44Z 2021-08-24T23:33:44Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 [Enhancement] Please allow 'insert-files' to insert content as text. 976399638  
905022931 https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905022931 https://api.github.com/repos/simonw/sqlite-utils/issues/321 IC_kwDOCGYnMM418Y3T codecov[bot] 22429695 2021-08-24T22:38:38Z 2021-08-24T23:27:26Z NONE

Codecov Report

Merging #321 (db2dd6d) into main (9258f4b) will increase coverage by 0.16%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #321      +/-   ##
==========================================
+ Coverage   96.41%   96.58%   +0.16%     
==========================================
  Files           5        5              
  Lines        2206     2223      +17     
==========================================
+ Hits         2127     2147      +20     
+ Misses         79       76       -3
<table> <thead> <tr> <th>Impacted Files</th> <th>Coverage Δ</th> <th></th> </tr> </thead> <tbody> <tr> <td>sqlite_utils/cli.py</td> <td>95.69% <100.00%> (+0.42%)</td> <td>:arrow_up:</td> </tr> </tbody> </table>

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9258f4b...db2dd6d. Read the comment docs.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Ability to insert file contents as text, in addition to blob 978537855  
905040902 https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905040902 https://api.github.com/repos/simonw/sqlite-utils/issues/321 IC_kwDOCGYnMM418dQG simonw 9599 2021-08-24T23:25:03Z 2021-08-24T23:25:03Z OWNER

I'm going to skip this test on windows.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Ability to insert file contents as text, in addition to blob 978537855  
905040307 https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905040307 https://api.github.com/repos/simonw/sqlite-utils/issues/321 IC_kwDOCGYnMM418dGz simonw 9599 2021-08-24T23:23:36Z 2021-08-24T23:23:36Z OWNER

https://discuss.python.org/t/pep-597-use-utf-8-for-default-text-file-encoding/1819 says:

Currently, TextIOWrapper uses locale.getpreferredencoding(False) (hereinafter called “locale encoding”) when encoding is not specified.
...
Package authors using macOS or Linux may forget that the default encoding
is not always UTF-8.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Ability to insert file contents as text, in addition to blob 978537855  
905039576 https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905039576 https://api.github.com/repos/simonw/sqlite-utils/issues/321 IC_kwDOCGYnMM418c7Y simonw 9599 2021-08-24T23:21:29Z 2021-08-24T23:21:29Z OWNER

Hah, the error here is actually:

>           assert result.exit_code == 1, result.output
E           AssertionError: 
E             
E           assert 0 == 1
E            +  where 0 = <Result okay>.exit_code

So I was expecting an error, but instead the command worked.

I suspect this is because on Windows the default character set may not be UTF-8?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Ability to insert file contents as text, in addition to blob 978537855  
905037323 https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905037323 https://api.github.com/repos/simonw/sqlite-utils/issues/321 IC_kwDOCGYnMM418cYL simonw 9599 2021-08-24T23:15:29Z 2021-08-24T23:15:29Z OWNER

Huh, tests are failing but only on Windows!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Ability to insert file contents as text, in addition to blob 978537855  
905024066 https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905024066 https://api.github.com/repos/simonw/sqlite-utils/issues/319 IC_kwDOCGYnMM418ZJC pjamargh 66709385 2021-08-24T22:41:39Z 2021-08-24T22:41:39Z NONE

I'm happy with this functionality left the way you describe. In my case the data is homogeneous but other cases would work just by being consistent on the encoding. Thanks a lot, Simon!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 [Enhancement] Please allow 'insert-files' to insert content as text. 976399638  
905021933 https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905021933 https://api.github.com/repos/simonw/sqlite-utils/issues/319 IC_kwDOCGYnMM418Ynt simonw 9599 2021-08-24T22:36:04Z 2021-08-24T22:36:04Z OWNER

Oh, I misread. Yes some files will not be valid UTF-8, I'd throw a warning and continue (not adding that file) but if you want to get more elaborate you could allow to define a policy on what to do. Not adding the file, index binary content or use a conversion policy like the ones available on Python's decode.

I thought about supporting those different policies (with something like --errors ignore) but I feel like that's getting a little bit too deep into the weeds. Right now if you try to import an invalid file the behaviour is the same as for the sqlite-utils insert command (I added the same detailed error message):

Error: Could not read file '/Users/simon/Dropbox/Development/sqlite-utils/data.txt' as text

'utf-8' codec can't decode byte 0xe3 in position 83: invalid continuation byte

The input you provided uses a character encoding other than utf-8.

You can fix this by passing the --encoding= option with the encoding of the file.

If you do not know the encoding, running 'file filename.csv' may tell you.

It's often worth trying: --encoding=latin-1

If someone has data that can't be translated to valid text using a known encoding, I'm happy leaving them to have to insert it into a BLOB column instead.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 [Enhancement] Please allow 'insert-files' to insert content as text. 976399638  
905021047 https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905021047 https://api.github.com/repos/simonw/sqlite-utils/issues/319 IC_kwDOCGYnMM418YZ3 simonw 9599 2021-08-24T22:33:48Z 2021-08-24T22:33:48Z OWNER

I had a few doubts about the design just now. Since content_text is supported as a special argument, an alternative way of handling the above would be:

sqlite-utils insert-files /tmp/text.db files *.txt -c path -c content_text -c size

This does exactly the same thing as just using --text and not specifying any columns, because the actual implementation of --text is as follows:

https://github.com/simonw/sqlite-utils/blob/0c796cd945b146b7395ff5f553861400be503867/sqlite_utils/cli.py#L1851-L1855

But actually I think that's OK - --text is a useful shorthand that avoids you having to remember how to manually specify those columns with -c. So I'm going to leave the design as is.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 [Enhancement] Please allow 'insert-files' to insert content as text. 976399638  
905021010 https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905021010 https://api.github.com/repos/simonw/sqlite-utils/issues/319 IC_kwDOCGYnMM418YZS pjamargh 66709385 2021-08-24T22:33:42Z 2021-08-24T22:33:42Z NONE

Oh, I misread. Yes some files will not be valid UTF-8, I'd throw a warning and continue (not adding that file) but if you want to get more elaborate you could allow to define a policy on what to do. Not adding the file, index binary content or use a conversion policy like the ones available on Python's decode.

From https://stackoverflow.com/questions/24616678/unicodedecodeerror-in-python-when-reading-a-file-how-to-ignore-the-error-and-ju :
- 'ignore' ignores errors. Note that ignoring encoding errors can lead to data loss.
- 'replace' causes a replacement marker (such as '?') to be inserted where there is malformed data.
- 'surrogateescape' will represent any incorrect bytes as code points in the Unicode Private Use Area ranging from U+DC80 to U+DCFF. These private code points will then be turned back into the same bytes when the surrogateescape error handler is used when writing data. This is useful for processing files in an unknown encoding.
- 'xmlcharrefreplace' is only supported when writing to a file. Characters not supported by the encoding are replaced with the appropriate XML character reference &#nnn;.
- 'backslashreplace' (also only supported when writing) replaces unsupported characters with Python’s backslashed escape sequences.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 [Enhancement] Please allow 'insert-files' to insert content as text. 976399638  
905013183 https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905013183 https://api.github.com/repos/simonw/sqlite-utils/issues/319 IC_kwDOCGYnMM418We_ simonw 9599 2021-08-24T22:15:34Z 2021-08-24T22:15:34Z OWNER

Here's the error message I have working for invalid unicode:

sqlite-utils insert-files /tmp/text.db files *.txt --text
  [------------------------------------]    0%
Error: Could not read file '/Users/simon/Dropbox/Development/sqlite-utils/data.txt' as text

'utf-8' codec can't decode byte 0xe3 in position 83: invalid continuation byte

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 [Enhancement] Please allow 'insert-files' to insert content as text. 976399638  
905013162 https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905013162 https://api.github.com/repos/simonw/sqlite-utils/issues/319 IC_kwDOCGYnMM418Weq simonw 9599 2021-08-24T22:15:31Z 2021-08-24T22:15:31Z OWNER

I'm going to assume utf-8 but allow --encoding to be used to specify something different, since that option is already supported by other commands.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 [Enhancement] Please allow 'insert-files' to insert content as text. 976399638  
905003381 https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905003381 https://api.github.com/repos/simonw/sqlite-utils/issues/319 IC_kwDOCGYnMM418UF1 pjamargh 66709385 2021-08-24T21:56:49Z 2021-08-24T21:56:49Z NONE

I was thinking that an approach could be making FILE_COLUMNS a generator (_get_file_columns(mode)) or you can just have a different set of columns (is there something else that makes sense to be changed on the text scenario?).

About UTF-8 I was referring to the encoding to use when reading files. This can be difficult to auto-detect but I believe that UTF-8 is pretty much the standard for text files.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 [Enhancement] Please allow 'insert-files' to insert content as text. 976399638  
905001586 https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905001586 https://api.github.com/repos/simonw/sqlite-utils/issues/319 IC_kwDOCGYnMM418Tpy simonw 9599 2021-08-24T21:52:50Z 2021-08-24T21:52:50Z OWNER

Will need to re-title this section of the documentation: https://sqlite-utils.datasette.io/en/3.16/cli.html#inserting-binary-data-from-files - "Inserting binary data from files" will become "Inserting data from files"

I'm OK with keeping the default as BLOB but I could add a --text option which stores the content as text instead.

If the text can't be stored as utf-8 I'll probably raise an error.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 [Enhancement] Please allow 'insert-files' to insert content as text. 976399638  
904999850 https://github.com/simonw/sqlite-utils/issues/319#issuecomment-904999850 https://api.github.com/repos/simonw/sqlite-utils/issues/319 IC_kwDOCGYnMM418TOq simonw 9599 2021-08-24T21:49:08Z 2021-08-24T21:49:08Z OWNER

This is a good idea.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 [Enhancement] Please allow 'insert-files' to insert content as text. 976399638  
904982056 https://github.com/simonw/datasette/issues/859#issuecomment-904982056 https://api.github.com/repos/simonw/datasette/issues/859 IC_kwDOBm6k_c418O4o brandonrobertz 2670795 2021-08-24T21:15:04Z 2021-08-24T21:15:30Z CONTRIBUTOR

I'm running into issues with this as well. All other pages seem to work with lots of DBs except the home page, which absolutely tanks. Would be willing to put some work into this, if there's been any kind of progress on concepts on how this ought to work.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Database page loads too slowly with many large tables (due to table counts) 642572841  
904954530 https://github.com/simonw/datasette/issues/1446#issuecomment-904954530 https://api.github.com/repos/simonw/datasette/issues/1446 IC_kwDOBm6k_c418IKi simonw 9599 2021-08-24T20:32:47Z 2021-08-24T20:32:47Z OWNER

Pasting that CSS into the styles editor in the developer tools on https://latest.datasette.io/ has the desired effect: footer at the bottom of the window unless the page is too long, in which case the footer is at the bottom of the scroll.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Modify base.html template to support optional sticky footer 978357984  
904866495 https://github.com/simonw/datasette/issues/1446#issuecomment-904866495 https://api.github.com/repos/simonw/datasette/issues/1446 IC_kwDOBm6k_c417yq_ simonw 9599 2021-08-24T18:13:49Z 2021-08-24T18:13:49Z OWNER

OK, now the following optional CSS gives us a sticky footer:

html, body {
  height: 100%;
}
body {
  display: flex;
  flex-direction: column;
}
.not-footer {
  flex: 1 0 auto;
}
footer {
  flex-shrink: 0;
}

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Modify base.html template to support optional sticky footer 978357984  
904642396 https://github.com/dogsheep/healthkit-to-sqlite/pull/13#issuecomment-904642396 https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/13 IC_kwDOC8tyDs41679c FabianHertwig 32016596 2021-08-24T13:27:40Z 2021-08-24T13:28:26Z NONE

This would fix #21 and make #22 obsolete.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 SQLite does not have case sensitive columns 743071410  

Next page

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);