Subject: [mongodb-user] getting writeError
massive insert files to mongoDB with gridfs
A have a python script witch takes a huge list of images and move them in a circle to mongoDB using gridfs (it stores files in two mongoDB collections, files and chunks).After few thousands of successful transfers I am starting to get following error:
I tried to debug it and it looks, that it becomes fail during chunk insert, but why this happens I don't know. I test script on my local mongoDB and here is a code on python, which I use
from pymongo import MongoClient
fs = gridfs.GridFS(db)
for image in images_list:
with open(vers['path'], 'rb') as fileObj:
grid_in = fs.new_file(filename=image['name'],
file_id = grid_in._id
except pymongo.errors.WriteError as we:
print('%s %s:%s' % (image['_id'], we, vers['path']))
I also tried gridfs.put instead of new_file and write and also GridFSBucket.open_upload_stream and GridfsBuscket.upload_from_stream - result always the same.
Since some moment I am getting error, which a mentioned above
Do you know why this happens and how can I solve it?
Or maybe you can at least answer following question:
- can be some problems because of inserting the same images few times? or images with same filename?
- can it be error because of access to mongoDB (lock mechanizm or smth) and how can I then change it to avoid any conflicts?
If you have some other ideas, they all are welcome.
You received this message because you are subscribed to the Google Groups "mongodb-user"
For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]
To post to this group, send email to [email protected]
Visit this group at https://groups.google.com/group/mongodb-user
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/797c142a-1015-4fe0-82bb-a51ec8ad22ff%40googlegroups.com
For more options, visit https://groups.google.com/d/optout