issue_comments

21 rows where user = 82988 sorted by updated_at descending

View and edit SQL

Suggested facets: issue_url, created_at (date), updated_at (date)

author_association

user

  • psychemedia · 21
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue
604328163 https://github.com/simonw/datasette/issues/573#issuecomment-604328163 https://api.github.com/repos/simonw/datasette/issues/573 MDEyOklzc3VlQ29tbWVudDYwNDMyODE2Mw== psychemedia 82988 2020-03-26T09:41:30Z 2020-03-26T09:41:30Z CONTRIBUTOR

Fixed by @simonw; example here: https://github.com/simonw/jupyterserverproxy-datasette-demo

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Exposing Datasette via Jupyter-server-proxy 492153532
586599424 https://github.com/simonw/datasette/issues/417#issuecomment-586599424 https://api.github.com/repos/simonw/datasette/issues/417 MDEyOklzc3VlQ29tbWVudDU4NjU5OTQyNA== psychemedia 82988 2020-02-15T15:12:19Z 2020-02-15T15:12:33Z CONTRIBUTOR

So could the polling support also allow you to call sqlite_utils to update a database with csv files? (Though I'm guessing you would only want to handle changed files? Do your scrapers check and cache csv datestamps/hashes?)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Datasette Library 421546944
580745213 https://github.com/simonw/sqlite-utils/issues/73#issuecomment-580745213 https://api.github.com/repos/simonw/sqlite-utils/issues/73 MDEyOklzc3VlQ29tbWVudDU4MDc0NTIxMw== psychemedia 82988 2020-01-31T14:02:38Z 2020-01-31T14:21:09Z NONE

So the conundrum continues.. The simple test case above now runs, but if I upsert a large number of new records (successfully) and then try to upsert a fewer number of new records to a different table, I get the same error.

If I run the same upserts again (which in the first case means there are no new records to add, because they were already added), the second upsert works correctly.

It feels as if the number of items added via an upsert >> the number of items I try to add in an upsert immediately after, I get the error.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
upsert_all() throws issue when upserting to empty table 545407916
573047321 https://github.com/simonw/sqlite-utils/issues/73#issuecomment-573047321 https://api.github.com/repos/simonw/sqlite-utils/issues/73 MDEyOklzc3VlQ29tbWVudDU3MzA0NzMyMQ== psychemedia 82988 2020-01-10T14:02:56Z 2020-01-10T14:09:23Z NONE

Hmmm... just tried with installs from pip and the repo (v2.0.0 and v2.0.1) and I get the error each time (start of second run through the second loop).

Could it be sqlite3? I'm on 3.30.1.

UPDATE: just tried it on jupyter.org/try and I get the error there, too.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
upsert_all() throws issue when upserting to empty table 545407916
571138093 https://github.com/simonw/sqlite-utils/issues/73#issuecomment-571138093 https://api.github.com/repos/simonw/sqlite-utils/issues/73 MDEyOklzc3VlQ29tbWVudDU3MTEzODA5Mw== psychemedia 82988 2020-01-06T13:28:31Z 2020-01-06T13:28:31Z NONE

I think I actually had several issues in play...

The missing key was one, but I think there is also an issue as per below.

For example, in the following:

def init_testdb(dbname='test.db'):

    if os.path.exists(dbname):
        os.remove(dbname)

    conn = sqlite3.connect(dbname)
    db = Database(conn)

    return conn, db

conn, db = init_testdb()

c = conn.cursor()
c.executescript('CREATE TABLE "test1" ("Col1" TEXT, "Col2" TEXT, PRIMARY KEY ("Col1"));')
c.executescript('CREATE TABLE "test2" ("Col1" TEXT, "Col2" TEXT, PRIMARY KEY ("Col1"));')

print('Test 1...')
for i in range(3):
    db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1'))
    db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1'))

print('Test 2...')
for i in range(3):
    db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1'))
    db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'},
                            {'Col1':'c','Col2':'x'}], pk=('Col1'))
print('Done...')

---------------------------------------------------------------------------
Test 1...
Test 2...
 IndexError: list index out of range 
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-763-444132ca189f> in <module>
     22 print('Test 2...')
     23 for i in range(3):
---> 24     db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1'))
     25     db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'},
     26                             {'Col1':'c','Col2':'x'}], pk=('Col1'))

/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in upsert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, extracts)
   1157             alter=alter,
   1158             extracts=extracts,
-> 1159             upsert=True,
   1160         )
   1161 

/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, ignore, replace, extracts, upsert)
   1097                 # self.last_rowid will be 0 if a "INSERT OR IGNORE" happened
   1098                 if (hash_id or pk) and self.last_rowid:
-> 1099                     row = list(self.rows_where("rowid = ?", [self.last_rowid]))[0]
   1100                     if hash_id:
   1101                         self.last_pk = row[hash_id]

IndexError: list index out of range

the first test works but the second fails. Is the length of the list of items being upserted leaking somewhere?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
upsert_all() throws issue when upserting to empty table 545407916
559632608 https://github.com/simonw/datasette/issues/573#issuecomment-559632608 https://api.github.com/repos/simonw/datasette/issues/573 MDEyOklzc3VlQ29tbWVudDU1OTYzMjYwOA== psychemedia 82988 2019-11-29T01:43:38Z 2019-11-29T01:43:38Z CONTRIBUTOR

In passing, it looks like a start was made on a datasette Jupyter server extension in https://github.com/lucasdurand/jupyter-datasette although the build fails in MyBinder.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Exposing Datasette via Jupyter-server-proxy 492153532
559207224 https://github.com/simonw/datasette/issues/642#issuecomment-559207224 https://api.github.com/repos/simonw/datasette/issues/642 MDEyOklzc3VlQ29tbWVudDU1OTIwNzIyNA== psychemedia 82988 2019-11-27T18:40:57Z 2019-11-27T18:41:07Z CONTRIBUTOR

Would cookie cutter approaches also work for creating various flavours of customised templates?

I need to try to create a couple of sites for myself to get a feel for what sorts of thing are easily doable, and what cribbable cookie cutter items might be. I'm guessing https://simonwillison.net/2019/Nov/25/niche-museums/ is a good place to start from?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Provide a cookiecutter template for creating new plugins 529429214
509013413 https://github.com/simonw/datasette/issues/507#issuecomment-509013413 https://api.github.com/repos/simonw/datasette/issues/507 MDEyOklzc3VlQ29tbWVudDUwOTAxMzQxMw== psychemedia 82988 2019-07-07T16:31:57Z 2019-07-07T16:31:57Z CONTRIBUTOR

Chrome and Firefox both support headless screengrabs from command line, but I don't know how parameterised they can be?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Every datasette plugin on the ecosystem page should have a screenshot 455852801
483202658 https://github.com/simonw/datasette/issues/429#issuecomment-483202658 https://api.github.com/repos/simonw/datasette/issues/429 MDEyOklzc3VlQ29tbWVudDQ4MzIwMjY1OA== psychemedia 82988 2019-04-15T10:48:01Z 2019-04-15T10:48:01Z CONTRIBUTOR

Minor UI observation:

_where= renders a [remove] link whereas _facet= gets a cross to remove it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_where=sql-fragment parameter for table views 432636432
483017176 https://github.com/simonw/datasette/issues/431#issuecomment-483017176 https://api.github.com/repos/simonw/datasette/issues/431 MDEyOklzc3VlQ29tbWVudDQ4MzAxNzE3Ng== psychemedia 82988 2019-04-14T16:58:37Z 2019-04-14T16:58:37Z CONTRIBUTOR

Hmm... nope... I see an updated timestamp from ls -al on the db but no reload?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Datasette doesn't reload when database file changes 432870248
482994231 https://github.com/simonw/sqlite-utils/issues/8#issuecomment-482994231 https://api.github.com/repos/simonw/sqlite-utils/issues/8 MDEyOklzc3VlQ29tbWVudDQ4Mjk5NDIzMQ== psychemedia 82988 2019-04-14T15:04:07Z 2019-04-14T15:29:33Z NONE

PLEASE IGNORE THE BELOW... I did a package update and rebuilt the kernel I was working in... may just have been an old version of sqlite_utils, seems to be working now. (Too many containers / too many environments!)

Has an issue been reintroduced here with FTS? eg I'm getting an error thrown by spaces in column names here:

/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order)

def enable_fts(self, columns, fts_version="FTS5"):
--> 329         "Enables FTS on the specified columns"
    330         sql = """
    331             CREATE VIRTUAL TABLE "{table}_fts" USING {fts_version} (

when trying an insert_all.

Also, if a col has a . in it, I seem to get:

/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order)
    327                     jsonify_if_needed(record.get(key, None)) for key in all_columns
    328                 )
--> 329             result = self.db.conn.execute(sql, values)
    330             self.db.conn.commit()
    331             self.last_id = result.lastrowid

OperationalError: near ".": syntax error

(Can't post a worked minimal example right now; racing trying to build something against a live timing screen that will stop until next weekend in an hour or two...)

PS Hmmm I did a test and they seem to work; I must be messing up s/where else...

import sqlite3
from sqlite_utils import Database

dbname='testingDB_sqlite_utils.db'

#!rm $dbname
conn = sqlite3.connect(dbname, timeout=10)


#Setup database tables
c = conn.cursor()

setup='''
CREATE TABLE IF NOT EXISTS  "test1" (
    "NO" INTEGER,
    "NAME" TEXT
);

CREATE TABLE IF NOT EXISTS  "test2" (
    "NO" INTEGER,
    `TIME OF DAY` TEXT
);

CREATE TABLE IF NOT EXISTS  "test3" (
    "NO" INTEGER,
    `AVG. SPEED (MPH)` FLOAT
);
'''

c.executescript(setup)


DB = Database(conn)

import pandas as pd

df1 = pd.DataFrame({'NO':[1,2],'NAME':['a','b']})
DB['test1'].insert_all(df1.to_dict(orient='records'))

df2 = pd.DataFrame({'NO':[1,2],'TIME OF DAY':['early on','late']})
DB['test2'].insert_all(df2.to_dict(orient='records'))

df3 = pd.DataFrame({'NO':[1,2],'AVG. SPEED (MPH)':['123.3','123.4']})
DB['test3'].insert_all(df3.to_dict(orient='records'))

all seem to work ok. I'm still getting errors in my set up though, which is not too different to the text cases?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Problems handling column names containing spaces or -  403922644
480621924 https://github.com/simonw/sqlite-utils/issues/18#issuecomment-480621924 https://api.github.com/repos/simonw/sqlite-utils/issues/18 MDEyOklzc3VlQ29tbWVudDQ4MDYyMTkyNA== psychemedia 82988 2019-04-07T19:31:42Z 2019-04-07T19:31:42Z NONE

I've just noticed that SQLite lets you IGNORE inserts that collide with a pre-existing key. This can be quite handy if you have a dataset that keeps changing in part, and you don't want to upsert and replace pre-existing PK rows but you do want to ignore collisions to existing PK rows.

Do sqlite_utils support such (cavalier!) behaviour?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.insert/.upsert/.insert_all/.upsert_all should add missing columns 413871266
474282321 https://github.com/simonw/datasette/issues/412#issuecomment-474282321 https://api.github.com/repos/simonw/datasette/issues/412 MDEyOklzc3VlQ29tbWVudDQ3NDI4MjMyMQ== psychemedia 82988 2019-03-19T10:09:46Z 2019-03-19T10:09:46Z CONTRIBUTOR

Does this also relate to https://github.com/simonw/datasette/issues/283 and the ability to ATTACH DATABASE?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Linked Data(sette) 411257981
474280581 https://github.com/simonw/datasette/issues/417#issuecomment-474280581 https://api.github.com/repos/simonw/datasette/issues/417 MDEyOklzc3VlQ29tbWVudDQ3NDI4MDU4MQ== psychemedia 82988 2019-03-19T10:06:42Z 2019-03-19T10:06:42Z CONTRIBUTOR

This would be really interesting but several possibilities in use arise, I think?

For example:

  • I put a new CSV file into the import dir and a new table is created therefrom
  • I put a CSV file into the import dir that replaces a previous file / table of the same name as a pre-existing table (eg files that contain monthly data in year to date). The data may also patch previous months, so a full replace / DROP on the original table may well be in order.
  • I put a CSV file into the import dir that updates a table of the same name as a pre-existing table (eg files that contain last month's data)

CSV files may also have messy names compared to the table you want. Or for an update CSV, may have the form MYTABLENAME-February2019.csv etc

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Datasette Library 421546944
464341721 https://github.com/simonw/sqlite-utils/issues/8#issuecomment-464341721 https://api.github.com/repos/simonw/sqlite-utils/issues/8 MDEyOklzc3VlQ29tbWVudDQ2NDM0MTcyMQ== psychemedia 82988 2019-02-16T12:08:41Z 2019-02-16T12:08:41Z NONE

We also get an error if a column name contains a .

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Problems handling column names containing spaces or -  403922644
459915995 https://github.com/simonw/datasette/issues/160#issuecomment-459915995 https://api.github.com/repos/simonw/datasette/issues/160 MDEyOklzc3VlQ29tbWVudDQ1OTkxNTk5NQ== psychemedia 82988 2019-02-02T00:43:16Z 2019-02-02T00:58:20Z CONTRIBUTOR

Do you have any simple working examples of how to use --static? Inspection of default served files suggests locations such as http://example.com/-/static/app.css?0e06ee.

If datasette is being proxied to http://example.com/foo/datasette, what form should arguments to --static take so that static files are correctly referenced?

Use case is here: https://github.com/psychemedia/jupyterserverproxy-datasette-demo Trying to do a really simple datasette demo in MyBinder using jupyter-server-proxy.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to bundle and serve additional static files 278208011
436042445 https://github.com/simonw/datasette/issues/370#issuecomment-436042445 https://api.github.com/repos/simonw/datasette/issues/370 MDEyOklzc3VlQ29tbWVudDQzNjA0MjQ0NQ== psychemedia 82988 2018-11-05T21:30:42Z 2018-11-05T21:31:48Z CONTRIBUTOR

Another route would be something like creating a datasette IPython magic for notebooks to take a dataframe and easily render it as a datasette. You'd need to run the app in the background rather than block execution in the notebook. Related to that, or to publishing a dataframe in notebook cell for use in other cells in a non-blocking way, there may be cribs in something like https://github.com/micahscopes/nbmultitask .

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Integration with JupyterLab 377155320
436037692 https://github.com/simonw/datasette/issues/370#issuecomment-436037692 https://api.github.com/repos/simonw/datasette/issues/370 MDEyOklzc3VlQ29tbWVudDQzNjAzNzY5Mg== psychemedia 82988 2018-11-05T21:15:47Z 2018-11-05T21:18:37Z CONTRIBUTOR

In terms of integration with pandas, I was pondering two different ways datasette/csvs_to_sqlite integration may work:

  • like pandasql, to provide a SQL query layer either by a direct connection to the sqlite db or via datasette API;
  • as an improvement of pandas.to_sql(), which is a bit ropey (e.g. pandas.to_sql_from_csvs(), routing the dataframe to sqlite via csvs_tosqlite rather than the dodgy mapping that pandas supports).

The pandas.publish_* idea could be quite interesting though... Would it be useful/fruitful to think about publish_ as a complement to pandas.to_?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Integration with JupyterLab 377155320
435862009 https://github.com/simonw/datasette/issues/371#issuecomment-435862009 https://api.github.com/repos/simonw/datasette/issues/371 MDEyOklzc3VlQ29tbWVudDQzNTg2MjAwOQ== psychemedia 82988 2018-11-05T12:48:35Z 2018-11-05T12:48:35Z CONTRIBUTOR

I think you need to register a domain name you own separately in order to get a non-IP address address? https://www.digitalocean.com/docs/networking/dns/

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette publish digitalocean plugin 377156339
401310732 https://github.com/simonw/datasette/issues/276#issuecomment-401310732 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDQwMTMxMDczMg== psychemedia 82988 2018-06-29T10:05:04Z 2018-06-29T10:07:25Z CONTRIBUTOR

@russs Different map projections can presumably be handled on the client side using a leaflet plugin to transform the geometry (eg kartena/Proj4Leaflet) although the leaflet side would need to detect or be informed of the original projection?

Another possibility would be to provide an easy way/guidance for users to create an FK'd table containing the WGS84 projection of a non-WGS84 geometry in the original/principle table? This could then as a proxy for serving GeoJSON to the leaflet map?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838
360535979 https://github.com/simonw/datasette/issues/179#issuecomment-360535979 https://api.github.com/repos/simonw/datasette/issues/179 MDEyOklzc3VlQ29tbWVudDM2MDUzNTk3OQ== psychemedia 82988 2018-01-25T17:18:24Z 2018-01-25T17:18:24Z CONTRIBUTOR

To summarise that thread:

  • expose full metadata.json object to the index page template, eg to allow tables to be referred to by name;
  • ability to import multiple metadata.json files, eg to allow metadata files created for a specific SQLite db to be reused in a datasette referring to several database files;

It could also be useful to allow users to import a python file containing custom functions that can that be loaded into scope and made available to custom templates.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
More metadata options for template authors  288438570

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);
Powered by Datasette · Query took 44.072ms · About: github-to-sqlite